acceptodds
Under review as a conference paper at ICLR 2027

Spec-CoFrGeNet: Reparametrizing Continued Fraction Networks for Stable and Accurate Training

Abstract

Continued fractions have been shown to be a function class capable of constructing expressive yet parameter-efficient neural networks. These neural networks are not only effective for supervised learning, but also for training large language models (LLMs). However, the rational form of this function class introduces poles when the denominator approaches zero, which prior work has avoided by enforcing a small positive (non-smooth) lower bound. We show that performance and training stability can be highly sensitive to the choice of this lower bound. In this work, we introduce a novel transformation of the partial denominators, inspired by its connection to the Green's function in physics, that mitigates the above issue elegantly by constraining the spectra of the constructed tridiagonal Hamiltonian. Our approach further eliminates the need for incremental training used by previous works for stability, thus avoiding involved modifications to standard pre-training procedures. We demonstrate the effectiveness of our reparametrization by replacing FFNs with a previously proposed continued fraction architecture for both language and image generative modeling. Across diverse models, such as GPT-2 XL, (nano) Llama-MoE, and Granite, we show that our proposed method achieves performance comparable to the original (FFN-based) architectures while achieving %-% parameter savings, and in most cases even outperforms the more complicated incremental training baseline. We further demonstrate, for the first time for continued fraction networks, that their effectiveness also carries over to image generative modeling, that utilize diffusion objectives. On a state-of-the-art Diffusion Transformer, we show that our approach again achieves performance comparable to the original FFN-based models with % parameter savings and significantly outperforms the previous incremental training approach.

Primary area:
generative models
Keywords:
continued fractions, generative AI

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3ihAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • ⁠This paper introduces a new approach to spectrally reparameterize continuous fraction networks, which avoids singularities and enables more stable training.
  • ⁠The authors replace feedforward layers in Transformers with this new class of layer, which avoids the need for incremental training, and run pertaining up to billions of parameters.
  • ⁠They demonstrate competitive performance in both language modelling and image generation.
Critical strengths
  • I like the fact that this paper arrives to the results through a theoretically motivated approach (similar to stability techniques used in digital signal processing), whose main advantages hold up empirically.
  • I appreciate the amount of experiments carried out by the authors, which gives value to their approach. The fact that the authors scaled their models to billions of parameters makes them particularly convincing.
  • I also like that the performance achieved stemmed from a significantly more parameter efficient approach, while simultaneously circumventing the incremental training approach used in previous work.
Critical weaknesses
  • I problem I see with the paper is the lack of an intuitive explanation of why continued fraction networks work well, or why they would be particularly suited to replace FFN layers in Transformers intuitively. I think this could be strengthened.
  • ⁠Although the authors have discussed the use of spectral constraints to prevent singularities, these do not necessarily ensure well behaved gradients. Transformer blocks rely on carefully designed normalization schemes and residual connections to effectively propagate information and avoid gradients issues in depth. It is not clear how the use of continued fraction networks would affect these properties.
Questions for authors
  • ⁠Is there an intuitive explanation for the performance of this type of network?
  • Have the authors investigated issues of gradient propagation with increased depth?
  • Is the network very sensitive the depth of the continued fraction used and the spectral constraints?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 3ihA (Part I)

We thank you for your constructive comments and appreciate the several positive remarks. Below, we address your major concerns. Please note that we have updated the paper where our edits are in pink and we point you to the relevant ones in our response below.

... the lack of an intuitive explanation of why continued fraction networks work well, or why they would be particularly suited to replace FFN layers in Transformers intuitively.

CoFrNets are a compact representation of a rich function class which enables them to express highly complex functional relationships and subsequently replace FFNs. Intuitively, a CoFrNet ladder of depth dd gives us a d+1d+1 degree rational function, while the number of parameters just scales linearly with depth. Thus we get a multiplicative representability benefit relative to the number of parameters required. In contrast, a FFN adds quadratic number of parameters per depth.

Moreover, having multiple ladders (i.e. width > 1) produces multiple rational functions which when combined into a single rational function would imply multiplying the denominator polynomials. This leads to multiplication of their degrees and hence a highly rich function. This idea was formally leveraged in the universal approximation proof for CoFrNets in Puri et. al. 2021 [1]. This is the reason for their favorable performance, while being parsimonious.

(a) Have the authors investigated issues of gradient propagation with increased depth?
(b) ⁠Although the authors have discussed the use of spectral constraints to prevent singularities, these do not necessarily ensure well behaved gradients. Transformer blocks rely on carefully designed normalization schemes and residual connections to effectively propagate information and avoid gradients issues in depth. It is not clear how the use of continued fraction networks would affect these properties.

First, we would like to mention that Spec-CoFrGeNet is robust to depth of a continued fraction ladder when replacing FFNs in language and vision models. Previous work [2] introduced the idea of incremental training i.e. freezing parameters at higher depths and then gradually training them in later iterations since, ladders of depth > 1 were unstable to train. However, with the reparametrization introduced in our work, we do not need such a stabilization scheme and are able to optimize deep ladders training everything together. This is evident from the experiments in our paper which have a (high) depth (and width) of 5.

Moreover, we provide additional results in the appendix for models using ladders with depth (and width) of 1. For language modeling, these include performance of pretrained models of GPT2-XL (Table A1), Granite 4.1 (Table A2) and Llama-MoE (Table A3). As observed from these results, the performance of models with the higher depth reported in the main paper were either similar or better. For image generation, we are also training DDT-XL/2 models using Spec-CoFrGeNet with depth, width of 1, 1 and 10, 10. As shown in Table A5, we observe that not only is there no stability issue with increasing depth, in fact some training losses are slightly lower indicating potential performance improvement with increased depth. Thus, there are no stability issues with increasing ladder depth with our reparametrization.

Second, we address the concern of gradient propagation through the network. We note that original layer normalizations and skip connections of the Transformer architectures are maintained, with only the FFNs being replaced with CoFrNets. This preserves the benefits of carefully established Transformer design choices, with our approach being a drop-in replacement and thus easy to integrate and evaluate.

Additionally, we now have performed a gradient analysis comparing CoFrGeNets and Spec-CoFrGeNets for vision where CoFrGeNets failed. The failure happens because the loss hits NaN values. We compute gradient norms for the available checkpoints just before this instability. Specifically, we compute ∣∣∂L/∂xl∣∣2||\partial\mathcal{L}/\partial x_l||_2, where L\mathcal{L} denotes the training loss and xlx_l denotes the activation after layer ll i.e. after residual addition. As observed in Fig. 5, CoFrGeNet has a gradient explosion of the order of 10510^5 through the backward pass which explains the observed instability. In contrast, the gradients of our proposed method stay stable.

We thank the reviewer for this thoughtful question. We have added the above analysis in Section 5.2 of our paper.

[1] Puri, Isha, et al. "CoFrNets: interpretable neural architecture inspired by continued fractions." Advances in neural information processing systems 34 (2021): 21668-21680.
[2] Dhurandhar, Amit, et al. "CoFrGeNet: Continued Fraction Architectures for Language Generation." ICML (2026).

AuthorsOpenReview ↗

Response to Reviewer 3ihA (Part II)

Is the network very sensitive the depth of the continued fraction used and the spectral constraints?

As mentioned above, the network is not sensitive to depth with our proposed reparametrization. In fact, performance overall improved slightly with depth.

Note that the spectral constraint is for the reparameterized partial denominators to lie outside the Gershgorin radius, which is exactly defined and computed. This guarantees stable results.

However, note that the above condition can be violated by setting δ\delta to a smaller value such as 0.00010.0001. In this scenario, δ\delta gets rounded to 00 due to bf16 precision, and this will lead to a singular tridiagonal matrix that makes training unstable. This however, reflects a data format limitation, rather than a methodological failure.

Finally, we also train a DDT-XL/2 model with a higher δ=0.01\delta=0.01 as shown in Table A6. We observe that the results for the higher δ\delta are only slightly worse in FID as expected, where training was stable. This further supports the robustness of our method and asserts the lack of need for tuning, which is highly beneficial given how expensive pre-training can be.

We have also now added some example language and image generations of Spec-CoFrGeNet in Appendix C.

AuthorsOpenReview ↗

Further questions?

Dear Reviewer 3ihA,

Since, the author-reviewer discussions are soon to close please let us know if there are any additional questions/concerns you might have. Thank you for your time and effort.

Reviewer_utp8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Continued-fraction networks express computation as nested ratios of learned functions, which blow up when a denominator approaches zero; prior work avoided this with a crude clamp plus a staged training schedule. This paper instead shifts the learned denominators so the blow-up cannot happen, removing the need for both remedies. Plugged into the feed-forward blocks of several language models and one image-diffusion model, it trains reliably and roughly matches the original models with substantially fewer parameters.

Critical strengths
  1. A real failure mode is identified and fixed. Under the diffusion objective the prior method fails to train at all (FID stuck around 565), while the proposed reparameterization trains normally. Making this work at all is the paper's most valuable result.
  2. Substantial experimental investment: GPT-2 XL on two corpora, a 2.14B Granite model, Llama-MoE, and DDT-XL/2 on ImageNet — with parameter-matched controls (Standard†).
  3. The fix is a one-line, hyperparameter-light change (a single fixed δ throughout) that is genuinely absent from the prior work.
Critical weaknesses
  1. The headline claim is misallocated. Parameter counts are identical between CoFrGeNet and Spec-CoFrGeNet in every table: the ~30% savings come from the prior work's architecture, not from this paper's method, yet the abstract presents them as its contribution. Likewise, "significantly outperforms the previous incremental training approach" holds only in diffusion, where the baseline fails to train outright — a rescue, not a win; on language tasks the deltas are below one point.
  2. The spectral/quantum framing is not used by the method. The final update is a diagonal-dominance shift (softplus plus a margin); no eigenvalues, Krylov subspaces, or related machinery appear anywhere in the actual algorithm. The presentation suggests more conceptual depth than the method carries.
  3. Missing ablations. The sign-preserving (PL) variant is introduced (Eq. 5, Fig. 3) and then never evaluated in any experiment; there is no sensitivity analysis for δ or ε beyond one loss curve, no depth/width ablation, and no measurement of how often poles would actually occur without the fix.
  4. Single seeds, small margins, one diffusion setting. Language gains are mostly under one point with no variance reported, and the diffusion result — the paper's strongest evidence — rests on a single backbone, dataset, and objective.
  5. A simpler alternative is not discussed. The paper's own baseline (CoFrNets) explicitly lists constraining the denominators to be positive as an alternative stabilization and rejects it as "restrictive as well as computationally challenging." This submission essentially adopts that route at scale, and should explain why the earlier concern does not apply — ideally with a direct comparison, since the constraint amounts to a few lines of change.
Questions for authors

Major:

  1. Why not compare against the simple positive-denominator constraint that your own baseline cites (a near-trivial modification of your model)? Relatedly, please clarify which of the claimed parameter savings originate in this work versus Dhurandhar et al. (2026).
  2. Evaluate the PL variant on at least one language and one vision setting; characterize the expressivity cost of forcing all partial denominators positive (this makes each block monotone in its inputs — is that a real restriction?); report 3 seeds for the flagship tables and sweep δ and ε.

Minor:

  1. Correct the accounting: Granite "outperforms in six out of eight" tasks is actually 5 wins / 1 tie / 2 losses versus Standard; the Llama-MoE BoolQ regression (60.8 → 53.7) and the datasets where Standard wins clearly (AgNews, LM1B) are unmentioned.
  2. What does the Green's-function/tridiagonal derivation add beyond the direct observation that positive denominators keep a continued fraction finite? If nothing operational, consider simplifying the framing.
  3. Release code ("available upon request" is insufficient); fix typos; state the fine-tuning learning-rate selection protocol.
5 comments
AuthorsOpenReview ↗

Response to Reviewer utp8 (Part I)

We thank you for your constructive comments and appreciate the positive remarks. Below, we address your major concerns. Please note that we have updated the paper where our edits are in pink and we point you to the relevant ones in our response below.

Why not compare against the simple positive-denominator constraint that your own baseline cites ?

Please note that the positive parameterization was discussed but not implemented in Puri et. al. 2021 [1].

The following reasons support the simple positive-denominator constraint to be an ineffective baseline:

  • A crucial issue is that just having a positive denominator does not guarantee that the continued fraction will not hit poles and subsequently does not prevent instability. An easy example are denominators approaching zero, albeit from the positive side. The spectral view provides a sufficient condition for the continued fractions to never hit their poles as explained in Section 4.
  • A positive-denominator constraint restricts the continued fraction ladders to be strictly positive while, our reparametrization scheme allows for negative as well as positive values. This is evident from f(a)f(a) with our reparameterizations in Section 4, thus giving greater expressivity.
  • Finally, the positive parameterization suggested in prior works was non-smooth (i.e. taking absolute values) which can be challenging for optimization as opposed to the preferred SPD parametrization suggested in this paper (see last three lines of Section 4).
  • Moreover, 11 was added to the absolute values in these prior works (Molina et. al. 2020 [2]) which can be interpreted as a high ϵ=1\epsilon=1. Our CoFrGeNet baseline used ϵ=0.1\epsilon=0.1 which makes it a much stronger baseline than the suggested strategy as it was sufficient for stable training for language modeling. Higher ϵ\epsilon values constrain the continued fractions more and limit expressibility, thus the lowest value that leads to stable training is preferred.

To further support our above claims, we tried two variants for enforcing positive denominators:

  • apply softplus to wTxw^Tx in each continued fraction ladder i.e. ai(x)=softplus(wiTx)+δa_i(x) = \text{softplus}(w^T_ix)+\delta
  • apply softplus to the entire continuant denominator i.e. f(a0,a)=Kd+1(a0,…,ad)softplus(Kd(a1,…,ad))+δf(a_0,a)=\frac{K_{d+1}(a_0,\ldots,a_d)}{\text{softplus}(K_d(a_1,\ldots,a_d))+\delta} (Please check Eq. 1 from the paper.) where, δ=0.001\delta=0.001 as for our spectral variants.

For language modeling, both led to unstable training where the validation loss reached NaNs. This is apparent because even though the denominators are positive, they got very close to δ\delta for certain inputs. We saw a similar trend while training vision models for image generation. We have now mentioned this in Section 5 of the paper.

please clarify which of the claimed parameter savings originate in this work versus Dhurandhar et al. (2026)

Our claim is that parameter savings over the original models are maintained in this work. Previous work [3] demonstrated parameter savings with CoFrGeNet for language modeling, however they needed incremental training to maintain stability during optimization. This entailed freezing parameters at higher depths and then gradually training them in later iterations. Additionally, ϵ\epsilon acted as another stability hyperparameter. With this context, we want to highlight the following:

  • If we change the parameterization, it is not obvious that the advantages (i.e. parameter savings) will follow. We show that not only is this maintained, rather we do not need the additional stabilizing mechanisms.
  • The previous work did not show any results for image generation. In fact, we show that CoFrGeNet becomes unstable when training DDT models. Our method not only achieves good performance, but also has the same parameter savings which is not the case for CoFrGeNet.

We believe the above to be significant contributions and hence we stated the same in the abstract.

However, following your review, we have clarified in the abstract that we maintain the parameter efficiency as opposed to it being a new phenomenon with the following phrasing: “...maintaining ∼\sim 30%-50% parameter savings as before…”. We thank you for helping us to clarify this.

The claim "significantly outperforms the previous incremental training approach" appears only in the last sentence of the abstract and is attributed only to the diffusion models, which we believe is accurate.

[1] Puri, Isha, et al. "CoFrNets: interpretable neural architecture inspired by continued fractions." Advances in neural information processing systems 34 (2021): 21668-21680.
[2] Molina, Alejandro, Patrick Schramowski, and Kristian Kersting. "Pad'e activation units: End-to-end learning of flexible activation functions in deep networks." arXiv preprint arXiv:1907.06732 (2019).
[3] Dhurandhar, Amit, et al. "CoFrGeNet: Continued Fraction Architectures for Language Generation." ICML (2026).

AuthorsOpenReview ↗

Response to Reviewer utp8 (Part II)

Evaluate the PL variant

We performed experiments with the PL reparametrization on Granite 4.1 for language modeling. Unlike the SPD reparametrization, the training wasn’t converging for depth and width 1. We believe this is due to the non-smooth nature of the parametrization.

However, training was stable for depth and width 5 and we report these results in Table A4 in the appendix. These are similar to the SPD reparametrization results reported in the main paper. We saw that with higher depths and widths, PL may becomes stable because the optimization reduces the weights for the ladders with exceedingly high activations and hence bases its predictions on stable well trained ladders.

sweep δ\delta and ϵ\epsilon

First, we want to distinguish the role of δ\delta and ϵ\epsilon in the two methods Spec-CoFrGeNet and CoFrGeNet respectively. ϵ\epsilon is a stabilising mechanism for CoFrGeNet which directly modulates the scale of the activations, where very small values can lead to arbitrarily small denominators resulting in poles. In contrast, δ\delta is not a stabilizing hyperparameter. The stabilization comes from the Gershgorin radius, as discussed in Section 4, while the delta is simply an offset, so that the zero eigenvalue do not lie at the edge of the Gershgorin disk. Thus, in principle, δ\delta can be arbitrarily small.

Noting this principle difference, the goal for both δ\delta and ϵ\epsilon is to identify the smallest value that leads to stable training, since smaller values constrain the corresponding models less. If a certain value leads to stable training, higher values will do the same, but further constrain the dynamic range leading to a similar or worse performance.

Now, as reported in the paper, for language modeling ϵ=0.1\epsilon=0.1 led to stable training while lower values like ϵ=0.01\epsilon=0.01 lead to poor convergence and numerical instability during training as showin in Fig. 1. For image generation, different ϵ\epsilon values like 0.01, 0.1 and 0.2 all lead to unstable training.

As mentioned before, δ\delta is simply an offset and so we choose a very small value of 1e−31e-3 which leads to desirable performance as demonstrated in the paper. Thus, we do not need to try higher values. Smaller values such as 0.00010.0001 get rounded to 00 due to bf16 precision. Note that this violates the sufficient condition in Section 4 from our paper, leading to a singular tridiagonal matrix that makes training unstable. Note this is a data format limitation, rather than a methodological failure.

Nonetheless to empirically verify our reasoning we trained a DDT-XL/2 model with a higher δ=0.01\delta=0.01 as shown in Table A6. We observe that the results for the higher δ\delta are only slightly worse in FID as expected, where training was stable. This further supports the robustness of our method and asserts the lack of need for tuning, which is highly beneficial given how expensive pre-training can be.

The spectral/quantum framing is not used by the method. The final update is a diagonal-dominance shift (softplus plus a margin); no eigenvalues, Krylov subspaces, or related machinery appear anywhere in the actual algorithm. The presentation suggests more conceptual depth than the method carries.

We would like to highlight the following points that support our framing of the manuscript:

  • We respectfully disagree that there are no eigenvalues used. Our method primarily hinges on having non zero eigenvalues to get a non-singular matrix as mentioned in Section 4. The parameterizations using the Gershgorin radius follows from this fact leading to the final update.
  • Note that the diagonal dominant shift is just one possible parametrization strategy based on the new spectral perspective that we have introduced. Requiring the tridiagonal matrix TT to be non-singular opens up the possibility of creating a whole class of CoFrGeNets with different reparametrizations. This goes beyond the SPD or PL reparameterizations which we have proposed in this work and can lead to models having other desirable properties (viz. robustness) beyond stable training. This possibility was not apparent from the previous works and is something that has emerged from the Green’s function-continued fraction connection.
  • Moreover, the diagonally dominant SPD reparametrization comes from writing the CoFrNet with subtractions in the partial denominators which was evident only from the Green’s function connection and wasn't observed a priori.

We have also now added some example language and image generations of Spec-CoFrGeNet in Appendix C.

AuthorsOpenReview ↗

Further questions?

Dear Reviewer utp8,

Since, the author-reviewer discussions are soon to close please let us know if there are any additional questions/concerns you might have. Thank you for your time and effort.

Reviewer_uTR3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Spec-CoFrGeNet, a spectral reparameterization of continued fraction networks to improve training stability while also removng need for incremental training. The method uses Gershgorin bounds to avoid poles and is evaluated on several language models and a diffusion transformer. Results are generally comparable to previous CoFrGeNet, with some improvements, particularly on diffusion

Critical strengths

[S1] Simple and potentially useful idea. The paper addresses a real training issue with continued fraction networks. The spectral reparameterization is mathematically motivated and relatively straightforward. Removing incremental training without sacrificing performance would certainly be useful.

[S2] Interesting diffusion results. I appreciate the experiments across different architectures. The diffusion results are particularly promising, since the previous CoFrGeNet reportedly fails to train, whereas the proposed method works reasonably well with fewer parameters.

Critical weaknesses

[W1] Limited novelty and significance. My main concern is that the contribution seems quite incremental over the recent CoFrGeNet paper (ICML 2026). CoFrNets have existed since 2021, and CoFrGeNet already introduced most of the architecture and training ideas for language generation. This paper mainly modifies the stabilization mechanism. While useful, its significance feels limited, especially since the broader impact of CoFrGeNet itself is still unclear. The extension to diffusion is interesting, but I'm not sure it alone makes the contribution sufficiently substantial. Most language results are also nearly identical to the predecessor.

[W2] Unclear necessity of the spectral formulation. The paper switches from an additive continued fraction in Eq. 1 to a subtractive one in Section 4, without really explaining why. This matters since their poles differ. Also, simpler positive-denominator parameterizations were already discussed in CoFrNets. It's unclear how much the Gershgorin constraints actually add over a basic softplus-based approach. I think this is an important missing baseline.

[W3] Empirical evidence could be stronger. Most improvements over CoFrGeNet are tiny, with no variance reported. The diffusion result is more interesting, but the previous method's failure is not analyzed in much detail. Also, the FID improvement over the equally sized standard FFN is only 1.626 to 1.606.

Questions for authors
  1. What would the authors consider the main conceptual advance beyond CoFrGeNet? Given that much of the architecture is inherited, what makes this a sufficiently significant contribution?

  2. Why does the continued fraction change signs between Eq. 1 and Section 4? Which version is actually implemented?

  3. How does the method compare to simply using softplus to ensure positive denominators, without the Gershgorin offsets?

  4. What's main rationale behind the diffusion failure of incremental CoFrGeNet?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uTR3 (Part I)

We thank you for your constructive comments and appreciate the positive remarks. Below, we address your major concerns. Please note that we have updated the paper where our edits are in pink and we point you to the relevant ones in our response below.

What would the authors consider the main conceptual advance beyond CoFrGeNet? Given that much of the architecture is inherited, what makes this a sufficiently significant contribution?

Our main contribution is providing a new perspective that opens up mechanisms to optimize CoFrGeNets across modalities in a robust and simple manner.

CoFrGeNets are a highly promising architecture class, since they perform on par or better than standard models with iso-parameters (Granite 4.1) and even with much fewer parameters (GPT2-XL, Llama-MoE). However, their optimization is dependent on careful tuning of ϵ\epsilon (as shown in Fig. 1) and incremental training, which adds non-trivial complexity to established pre-training protocols. Thus, enhancing the stability of the training and making it independent of such measures will, in our opinion, have measurable impact on their wide adoption. Specifically, we make the following main contributions:

  • In this work, we provide a novel way of effectively optimizing CoFrGeNets that works not only for language but also image generation. As we demonstate, CoFrGeNets cannot train models for image generation due to instability.
  • The fact that we do not have to sweep ϵ\epsilon anymore is a major practical advancement. All our models are pretrained i.e. trained from scratch which is an extremely expensive (computationally, monetarily, energy wise) exercise. Even performing tuning for 2 values of ϵ\epsilon can be extremely expensive. Hence, without having to tune such hyperparameters (viz. ϵ\epsilon), we are able to perform accurate and stable training of CoFrGeNets that give similar or better results on language and considerably better results on vision is we believe to be an important step forward. Moreover, we forgo the need for implementing the complicated incremental training procedure in pre-training.
  • In our method, we present a novel perspective requiring the tridiagonal matrix TT to be non-singular (Section 4). This opens up the possibility of creating a whole class of CoFrGeNets that respect this constraint going beyond the proposed SPD and PL reparameterizations. This can lead to the models having other desirable properties (viz. being robust) beyond stable training. This possibility was not apparent from previous works.

Why does the continued fraction change signs between Eq. 1 and Section 4? Which version is actually implemented?

The minus sign version mentioned in Section 4 is implemented.

Note that Eq.1 is a generic form of continued fraction used to introduce the concept. Changing the signs there would not change the representation power or expressibility.

Subsequently, we change to a negative sign to make it consistent with the Green’s function perspective presented in the paper. This enables us to obtain a symmetric tridiagonal matrix TT (in Section 4), which would not be symmetric otherwise. Additionally, the symmetric nature of this matrix enables us to get the SPD parameterization.

AuthorsOpenReview ↗

Response to Reviewer uTR3 (Part II)

(a) Simpler positive-denominator parameterizations were already discussed in CoFrNets. It's unclear how much the Gershgorin constraints actually add over a basic softplus-based approach. I think this is an important missing baseline.
(b) How does the method compare to simply using softplus to ensure positive denominators, without the Gershgorin offsets?

Please note that the positive parameterization was discussed but not implemented in Puri et. al. 2021 [1].

The following reasons support the simple positive-denominator constraint to be an ineffective baseline:

  • A crucial issue is that just having a positive denominator does not guarantee that the continued fraction will not hit poles and subsequently does not prevent instability. An easy example are denominators approaching zero, albeit from the positive side. The spectral view provides a sufficient condition for the continued fractions to never hit their poles as explained in Section 4.
  • A positive-denominator constraint restricts the continued fraction ladders to be strictly positive while, our reparametrization scheme allows for negative as well as positive values. This is evident from f(a)f(a) with our reparameterizations in Section 4, thus giving greater expressivity.
  • Finally, the positive parameterization suggested in prior works was non-smooth (i.e. taking absolute values) which can be challenging for optimization as opposed to the preferred SPD parametrization suggested in this paper (see last three lines of Section 4).
  • Moreover, 11 was added to the absolute values in these prior works (Molina et. al. 2020 [2]) which can be interpreted as a high ϵ=1\epsilon=1. Our CoFrGeNet baseline used ϵ=0.1\epsilon=0.1 which makes it a much stronger baseline than the suggested strategy as it was sufficient for stable training for language modeling. Higher ϵ\epsilon values constrain the continued fractions more and limit expressibility, thus the lowest value that leads to stable training is preferred.

To further support our above claims, we tried two variants for enforcing positive denominators:

  • apply softplus to wTxw^Tx in each continued fraction ladder i.e. ai(x)=softplus(wiTx)+δa_i(x) = \text{softplus}(w^T_ix)+\delta
  • apply softplus to the entire continuant denominator i.e. f(a0,a)=Kd+1(a0,…,ad)softplus(Kd(a1,…,ad))+δf(a_0,a) =\frac{K_{d+1}(a_0,\ldots,a_d)} {\text{softplus}(K_d(a_1,\ldots,a_d))+\delta} (Please check Eq. 1 from the paper.) where, δ=0.001\delta=0.001 as for our spectral variants.

For language modeling, both led to unstable training where the validation loss reached NaNs. This is because even though the denominators are positive, they got very close to δ\delta for some inputs. We saw a similar trend while training vision models for image generation. We have now mentioned this in Section 5 of the paper.

What's main rationale behind the diffusion failure of incremental CoFrGeNet?

To investigate the failure behind the incremental CoFrNet in vision models, we performed a gradient analysis comparing CoFrGeNets and Spec-CoFrGeNets.

Note that the failure happens because the loss hits NaN values. We computed gradient norms for the available checkpoints just before this instability. Specifically, we computed ∣∣∂L/∂xl∣∣2||\partial\mathcal{L}/\partial x_l||_2, where L\mathcal{L} denotes the training loss and xlx_l denotes the activation after layer ll i.e. after residual addition. The loss is computed as the mean loss from 100 randomly selected samples from the dataset.

As observed in Fig. 5, CoFrGeNet has a gradient explosion of the order of 10510^5 through the backward pass which explains the observed instability. In contrast, the gradients of our proposed method stay stable and show well-behaved gradients.

This explains the failure of using CoFrGeNet on DDT models. We thank you for this question. We have now added this analysis to Section 5.2 in the paper.

Most improvements over CoFrGeNet are tiny

Our main intent was not necessarily to propose a method that significantly outperformed CoFrGeNet on language but rather have the following benefits:

  • CoFrGeNets fail for diffusion models due to gradient explosion, whereas our method is stable.
  • Despite the new reparameterization, we maintain the parameter efficiency and performance while making the training more stable on language even without the complicated incremental training.
  • We do not need to sweep epsilon values, which is a major practical benefit given how expensive pre-training is.
  • Opens up possibilities for a class of CoFrGeNets.

We have also now added some example language and image generations of Spec-CoFrGeNet in Appendix C.

[1] Puri, Isha, et al. "CoFrNets: interpretable neural architecture inspired by continued fractions." Advances in neural information processing systems 34 (2021): 21668-21680.
[2] Molina, Alejandro, Patrick Schramowski, and Kristian Kersting. "Pad'e activation units: End-to-end learning of flexible activation functions in deep networks." arXiv preprint arXiv:1907.06732 (2019).

AuthorsOpenReview ↗

Further questions?

Dear Reviewer uTR3,

Since, the author-reviewer discussions are soon to close please let us know if there are any additional questions/concerns you might have. Thank you for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.