acceptodds
Under review as a conference paper at ICLR 2027

Variational Kolmogorov-Arnold Network

Abstract

Kolmogorov-Arnold Networks (KANs) offer a theoretically grounded alternative to multi-layer perceptrons by representing multivariate functions as compositions of univariate basis functions. However, a critical limitation of KANs is the need to manually specify the number of basis functions per layer–a hyperparameter that directly controls model capacity and substantially impacts performance, yet whose optimal value varies unpredictably across tasks. We present InfinityKAN, a variational inference framework that eliminates this design choice by learning the number of basis functions during training. Our approach models the basis count as a latent variable with a truncated exponential prior, introducing a differentiable weighting function that enables gradient-based optimization. We establish the Lipschitz continuity of the variational objective, ensuring stable training dynamics. Experiments across 18 datasets spanning synthetic, image, tabular, and graph domains demonstrate that InfinityKAN matches or exceeds the performance of KANs while requiring no manual selection of the number of bases for each layer.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Supervised Learning, Adaptive Architectures, Kolmogorov-Arnold Networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AFASOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to remove the hyperparameter responsible for selecting the number of basis functions to use in KANs. To do so they propose to use Variational Inference to dynamically learn it via backpropagation. They also solve the zero-gradient problem with soft weighting and develop a specific loss function.

Critical strengths
  1. I really like the idea of dynamically learning the number of basis functions to use and the tricks used to allow training it. This can eliminate an important KAN bottleneck.
  2. The paper is well written and easy to follow. The "generative AI tools [used] to assist with LaTeX formatting" were properly employed.
Critical weaknesses
  1. You are removing one hyperparameter and adding 3/4 of them. Despite you partially state that ("INFINITYKAN introduces two hyperparameters", why didn't you count τ\tau or the initial basis count?) state that, it seems that you effectively need more tuning. This, added to the fact that each run takes double the time makes me question the effectiveness of the proposed solution.
  2. The evaluations, despite being abundant, consider only very simple settings. For example you use very shallow networks and achieve only 47% accuracy on CIFAR-10.
Questions for authors
  1. In Appendix L you try to produce evidence on the fact that some of the added hyperparameters are not important. a) is it the case and based on what evidence? b) Why did you consider only those 2 hyperparameters?
  2. How does your method extend beyond "trivial" settings / datasets?
3 comments
AuthorsOpenReview ↗

Rebuttal by authors

(W1, Qa, Qb) Hyperparameter accounting.

We do not claim zero hyperparameters, and we have replaced the "two hyperparameters" phrasing with explicitly mentioning them. The quantities that are tuned for InfinityKAN are the two priors, η\eta on the rate and σ\sigma on the coefficients. The quantile threshold is fixed at τ=0.9\tau=0.9 for all 18 datasets and never tuned, as also done in previous works; the starting basis count was varied only on the three synthetic datasets and fixed elsewhere; the weighting distribution and the activation were selected once on the synthetic datasets (truncated exponential, PReLU) and reused unchanged. What is removed is the per-layer basis count, which in a fixed-KK KAN multiplies the rest of the grid (mm candidates per layer, mLm^L combinations in the worst case) and in our protocol accounted for about half of all configurations (App. K). On the evidence in App. L: Figs. 8 and 9 show, over 18 datasets and 10 seeds, that η\eta and σ\sigma mainly control the magnitude and the run-to-run variance of the learned basis counts, with η=10\eta=10 driving the model to about 2 bases and η=0\eta=0 leaving counts between 2 and 10. We agree this measures the effect on the learned architecture rather than on accuracy. We have adjusted the claim accordingly; App. N adds the asymptotic argument (Theorem N.1) for why prior influence decreases with sample size [1]. We report the two priors because they are the only new tuned quantities; the others are fixed by protocol, and we now say so explicitly.

[1] Bernardo, J. M. and Smith, A. F. Bayesian theory, volume, 405. John Wiley & Sons, 2009

(W1, continued) Effectiveness given the per-run cost.

We agree the comparison must be made where the cost actually is. The ≈2×\approx2\times per-run overhead is stated in the limitations, and it is specific to the small fully-connected setting, where resizing the coefficient tensors is a significant fraction of the step cost. In the convolutional setting added during the rebuttal, the per-run wall-clock of InfinityCKAN and CKAN is comparable because the convolutions dominate. At the pipeline level, Table 3 reports end-to-end times including model selection for both methods: InfinityKAN is faster on 13 of 18 datasets and more accurate on 10 of 18, with about half the configurations. We have revised the Model selection paragraph and the Table 3 caption to state this trade-off explicitly, including the regime where fixed-basis KAN remains preferable.

(W2, Qc) CIFAR-10 accuracy and beyond simple settings.

The 47% in Table 1 comes from a deliberately shallow, parameter-matched backbone shared by KAN, MLP and InfinityKAN, chosen for a controlled comparison rather than for absolute accuracy; within that setting InfinityKAN gives the largest relative gains on the hardest datasets (CIFAR10 43.00 to 46.99, CIFAR100 14.04 to 19.27). To address the concern directly, we added App. M.1 (Table 5), where all 8 layers of a convolutional KAN backbone adapt their basis count: InfinityKAN-Conv reaches 87.46±0.3987.46 \pm 0.39 test accuracy against 86.83±0.5186.83 \pm 0.51 for the fixed-basis counterpart whose basis count was selected by sweep, over 10 runs each, under an identical protocol and a budget matched at 4 configurations per method. A second and smaller convolutional backbone is the one behind the C(K)C(K) study of Fig. 11, at 80.88±0.1980.88 \pm 0.19; its fixed-basis counterpart is still training, so we label the convolutional results as partial and will complete them in the revision. Already at this stage, they show that the 47% follows from the shallow backbone rather than from the adaptive mechanism, and that basis adaptation keeps working when the KAN layers sit inside a convolutional architecture.

I thank the authors for the response. I will keep the score

We thank the reviewer for taking the time to review the rebuttal and the modified document.

We would like to ask whether there are still critical issues in our work that did not allow the reviewer to increase the score, and we would appreciate the Reviewer's guidance in this regard.

Thank you again for your efforts in reviewing our work.

Reviewer_eQicOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes InfinityKAN, a framework that learns the number of basis functions in each Kolmogorov–Arnold network layer during training, addressing the need for manual basis-count selection. It considers an unbounded family of basis expansions and introduces layer-specific latent rate parameters that determine finite basis counts through exponential-quantile truncation. Differentiable basis weights enable gradient-based updates of these rates, while parameter tensors are expanded or truncated as the selected counts change. Training uses a deterministic approximation to a variational objective, with priors on the coefficients and rate parameters. The paper also analyzes the objective’s regularity and the effects of weighting, truncation, and prior penalties. Experiments on classification datasets spanning synthetic, image, tabular, and graph domains compare InfinityKAN with fixed-basis KANs and MLPs, reporting predictive accuracy, learned capacity, training dynamics, and computational costs. Results show competitive performance, with improvements on several benchmarks.

Critical strengths

1. Clear motivation and methodological presentation.

The paper addresses a practical challenge in KANs: selecting the number of basis functions for each layer. Learning this quantity during training is a useful direction for reducing manual architecture selection. The graphical model and explicit training algorithm make the proposed variational formulation, basis weighting, and parameter updates easy to follow.

2. Broad empirical evaluation.

The evaluation covers 18 datasets across synthetic, image, tabular, and graph domains, with means and standard deviations reported over ten runs. Comparisons with fixed-basis KANs and MLPs demonstrate improvements on several benchmarks, particularly CIFAR10 and CIFAR100. The reported training curves also provide evidence that adaptive basis counts can stabilize during optimization.

Critical weaknesses

1. Problems with the variational objective and its justification.

The coefficient contribution in Equation (6) is

\left[\log p(\theta)-\log q(\theta\mid\nu)\right].$$ Under Equation (8), inactive parameter contributions cancel because their variational distributions equal their priors. For the remaining active parameters, the paper’s mean-evaluation approximation assigns each scalar parameter a contribution of $-\frac12\log(2\pi)$ to $\log q$. Although this value is independent of the parameter mean, its aggregate contribution depends on the learned basis counts. Equation (9) therefore cannot discard it as an architecture-independent constant. This affects the interpretation of the complexity penalty in Appendix D. Equation (21) correctly sums the Gaussian log-priors retained in Equation (9), but restoring the omitted contribution gives, in Theorem D.3’s notation, $$\sum_{k=1}^{K} \left[\log p(\bar\theta_k)-\log q(\bar\theta_k)\right] = -\frac K2\log\sigma^2 -\frac{1}{2\sigma^2}\sum_{k=1}^{K}\bar\theta_k^2.$$ In particular, the term depending only on the parameter count vanishes when $\sigma^2=1$. The stated Occam-penalty interpretation, and its use in Remark D.4 to explain why basis counts do not systematically increase, therefore require revision. Separately, replacing variational expectations with evaluations at the parameter means is an additional approximation beyond the mean-field assumption. This substitution does not generally preserve the lower-bound property, and its accuracy in the evaluated setting is not established. The resulting objective therefore requires stronger justification and is more appropriately described as a deterministic VI-inspired surrogate in its current form. **2. Limited evidence of practical benefit and differentiation from prior work.** Although InfinityKAN achieves competitive results, the experiments do not clearly establish when practitioners should prefer adaptive basis selection. Table 2 reports selected architectures and learned basis counts, but these alone do not isolate the benefit of adapting $K$, particularly when depth and hidden dimensions also differ between models. Table 3 addresses computational efficiency, but the total cost of tuning the additional hyperparameters and initial basis counts needs to be made explicit. Furthermore, AWNNs [1] already introduce a closely related mechanism for learning neuron counts through variational modeling and dynamic truncation. The paper does not sufficiently establish which KAN-specific challenges require new methodology or what advantages adapting basis counts provides over adapting network width. A controlled comparison would help clarify both the methodological contribution and its practical value. [1] Errica et al. Adaptive Width Neural Networks. ICLR, 2026.
Questions for authors
  1. What advantages does deterministic quantile-based truncation offer over directly modeling a discrete distribution over KK such as a shifted Poisson distribution?

  2. Could the authors clarify the methodological differences from AWNNs [1] and compare the benefits and limitations of adapting basis counts versus neuron counts?

  3. Could the authors clarify whether the cost comparison in Table 3 includes all model-selection costs, including the additional hyperparameters and initial basis counts? Under a matched total computational budget, when does InfinityKAN offer a better accuracy–cost trade-off than tuning a fixed-basis KAN?

[1] Errica et al. Adaptive Width Neural Networks. ICLR, 2026.

3 comments
AuthorsOpenReview ↗

Rebuttal by authors

(W1) Equation (9), the omitted constant, and the Occam interpretation.

The reviewer's derivation is correct, and we have adopted it. Eq. (9) now keeps the per-coefficient contribution 12ln⁡(2π)\tfrac12\ln(2\pi), so the parameter-side term is Rθ(K)+C(K)R_\theta(K)+C(K) with Rθ(K)=∑k≤Kln⁡p(θˉk)R_\theta(K)=\sum_{k\le K}\ln p(\bar\theta_k) and C(K)=n(K)2ln⁡(2π)C(K)=\tfrac{n(K)}{2}\ln(2\pi), n(K)=dℓdℓ−1Kn(K)=d_\ell d_{\ell-1}K (App. E.1). We also confirm the consequence the reviewer points out: since Rθ(K)+C(K)=−n(K)2ln⁡σ2−12σ2∑kθˉk2R_\theta(K)+C(K)=-\tfrac{n(K)}{2}\ln\sigma^{2}-\tfrac{1}{2\sigma^{2}}\sum_k\bar\theta_k^{2}, the term that depends on the parameter count alone vanishes at σ=1\sigma=1 and changes sign for σ<1\sigma<1.

Theorem D.3 and Remark D.4 are revised accordingly: the complexity is not the count-only penalty in general, but the coefficient-magnitude term together with the rate-prior term Rν(ν)=ln⁡p(ν;η)−ln⁡q(ν;νˉ)R_\nu(\nu)=\ln p(\nu;\eta)-\ln q(\nu;\bar\nu). The net effect depends on σ\sigma. The empirical counterpart is Fig. 11 (App. M), a controlled pair of convolutional CIFAR10 runs whose only difference is the inclusion of C(K)C(K): the learned basis-count trajectories coincide (16 bases in both) and validation accuracy differs by only about half a point; in this setting hence, including the term corrects the derivation without significantly altering the selected architecture. Finally, we now describe the training objective as a deterministic first-order ELBO approximation, with the gap bounded in Prop. E.1, rather than as the ELBO itself.

(Q1) Quantile truncation versus a discrete count distribution (e.g. shifted Poisson).

A discrete count prior (as shifted Poisson) is a legitimate alternative, but in practice it still requires finite-support truncation or a bounded-domain approximation to be trainable, so the truncation choice reappears in a different way; the trade-off is now stated in Sec. 3 ("The Weighting Function for the Basis").

(Q2) Difference from AWNN and what is KAN-specific.

AWNN adapts the number of neurons, i.e. the layer dimensions, which in a KAN corresponds to adapting dℓd_\ell. InfinityKAN keeps dℓd_\ell fixed and adapts KℓK_\ell, the cardinality of the basis expansion inside each edge function; the two axes are complementary and can in principle be combined. The KAN-specific difficulty is that capacity is added inside a univariate expansion rather than as a new additive unit: the output is not additive in the new component, so the analysis has to address approximation by increasing basis count (Theorems A.2, A.3), the deviation of soft weighting from hard truncation (Theorem D.1), and the ELBO jump at a basis-count transition (Theorem D.2) - none of which arise when neurons are appended additively. Related Work and Sec. 3 now state this explicitly. A fully controlled AWNN comparison would require choosing a width-adaptive KAN baseline and is a natural follow-up.

(Q3) Does Table 3 include all model-selection costs?

Yes, with the following accounting, now stated in the caption and in App. K. The shared grid (hidden size {2,5,16}\{2,5,16\}, depth {1,2}\{1,2\}, extended to {1,2,5}\{1,2,5\} on graphs) and the training protocol are identical for all methods. On top of it, KAN sweeps the basis count, while InfinityKAN sweeps the two priors; τ=0.9\tau=0.9 and the starting basis count are fixed. This makes KAN's total about twice InfinityKAN's in number of configurations. Under this budget, InfinityKAN is faster end-to-end on 13 of 18 datasets and more accurate on 10 of 18. The budget-matched convolutional experiment described in R1 (4 configurations per method, 87.46±0.3987.46 \pm 0.39 against 86.83±0.5186.83 \pm 0.51) is a direct test of the reviewer's question. Conversely, when a good fixed basis count is known in advance, a fixed-basis KAN remains competitive in wall-clock time; this regime dependence is now stated explicitly.

Thank you for the revisions and additional experiments. I appreciate the correction to Equation (9) and the clarification of the objective. I have two further comments:

  1. The correction to Equation (9) should also be reflected in Theorem D.2, whose jump bound still omits the restored per-coefficient contribution 12log⁡(2π)\frac12\log(2\pi).

  2. Although unbounded support can complicate computation, shifted-Poisson counts can in principle be trained using Monte Carlo score-function estimators under suitable regularity conditions [1]. Whether this approach would be practically preferable to deterministic quantile truncation remains unclear, given the trade-offs in gradient variance and computational cost, but it would be worth exploring.

My main remaining concern is that, given the structural similarities to AWNN [2], the response does not sufficiently clarify what additional methodological contribution arises from the different adaptation target and modeling choices. The added experiments strengthen the empirical evidence, but the benefit specifically attributable to dynamic basis-count adaptation remains unclear.

I therefore maintain my score.

[1] Schulman et al. Gradient Estimation Using Stochastic Computation Graphs. NeurIPS, 2015.

[2] Errica et al. Adaptive Width Neural Networks. ICLR, 2026.

We thank the reviewer for the pointer to score-function estimators, which we have followed in the revision (changes highlighted).

Score-function variant (new App. N). The reviewer is right that a discrete count prior does not need truncation to be trainable: with a score-function estimator [1], the count can follow an unbounded distribution (e.g. a shifted Poisson), and only the sampled, finite architecture is instantiated. Sec. 3 now presents this as a legitimate alternative, and App. N, which acknowledges the suggestion, gives:

  • Method and pseudo-code, for the truncated and the untruncated version: counts sampled at each step, pathwise gradient through the basis weights plus the score term (f(K)−b)∇ln⁡q(K∣ν)(f(K)-b)\nabla\ln q(K\mid\nu) with a moving-average baseline, nested bases with grow-only allocation.
  • Regularity. Lemma N.1 bounds the change of the objective between consecutive counts by 2Clik/(K+1)2C_{\mathrm{lik}}/(K+1). With an unbounded support, Theorem N.2 shows that the expected objective is smooth and Lipschitz in the rate, so the jumps of Theorems 3.1 and D.2 disappear; with a truncated support, jumps remain but are bounded by 2Clikln⁡(Kˉ+1)/(Kˉ+1)2C_{\mathrm{lik}}\ln(\bar K+1)/(\bar K+1).
  • Trade-offs: estimator variance, and memory, since without a cap the allocated size follows the largest count sampled. At inference the count is deterministic.

Occam term (Theorem D.3). Each added coefficient changes the parameter-side term of Eq. (9) by −12ln⁡σ2−θˉ2/(2σ2)-\tfrac12\ln\sigma^2-\bar\theta^2/(2\sigma^2), so a per-basis penalty arises only for σ>1\sigma>1; for σ≤1\sigma\le1, complexity is controlled by the coefficient magnitudes and the rate prior. This sign change comes from the first-order surrogate: with q(θ)=N(θˉ,1)q(\theta)=\mathcal N(\bar\theta,1), the exact expectation gives the count-only term 12(1−σ−2−ln⁡σ2)≤0\tfrac12(1-\sigma^{-2}-\ln\sigma^2)\le0 per coefficient, a penalty for every σ≠1\sigma\neq1. Since it is in closed form, a new remark (App. E.1) gives this objective, which differs from Eq. (9) by ∑ℓnℓ2(1−σℓ−2)\sum_\ell\tfrac{n_\ell}{2}(1-\sigma_\ell^{-2}) and has the same coefficient gradients.

Ablations (new App. L, Table 5, and App. N.1, Table 8). On Phoneme, with four layers starting from 16 bases each and σ=η=10\sigma=\eta=10, we varied the count-only term (none, Eq. (9), or exact, evaluated on the continuous count so that it has a gradient in the rate) and how the count is trained (deterministic; sampled on the truncated support; or sampled as 1+Poisson(λℓ)1+\mathrm{Poisson}(\lambda_\ell), with a separate rate and no cap). Mean (std) over 10 runs, 3 for the unbounded variant:

TrainingCount termVal. acc.Test acc.Bases
deterministicnone90.18 (0.85)87.44 (0.59)9.9 (1.8)
deterministicEq. (9)89.49 (0.94)86.71 (0.69)8.0 (0.0)
deterministicexact90.51 (0.38)86.40 (0.80)8.0 (0.0)
sampled, truncatednone89.40 (0.82)86.59 (0.96)16.8 (15.2)
sampled, truncatedexact89.35 (1.18)86.65 (0.72)8.1 (0.3)
sampled, unboundedexact89.61 (1.24)86.71 (1.00)14.7 (15.1)
  • The count-only term drives the count: with either form, all layers reach the minimum of 2 bases within 14 epochs, while without it the count decreases gradually. The first-order and exact forms give identical trajectories; at σ=10\sigma=10 both are negative, and they differ in sign only for σ<1\sigma<1, which is not tested here.
  • Accuracy differences are about one standard deviation and are not consistent between validation and test, so the variants can be considered equivalent in accuracy.
  • Sampling does not improve accuracy and costs 2 to 4 times the training time. Without the count term, truncated sampling is bimodal (7 of 10 runs reach the minimum). With the unbounded support, the count reaches one basis per layer by epoch 250 in all runs (the spread reflects one checkpoint selected at epoch 82), but large early samples allocate 27 to 38 bases per layer.

We therefore keep the deterministic surrogate as the default, while the unbounded variant remains of interest for the smoothness of its expected objective. Apart from the sampled-count comparison, all experiments use the deterministic truncation.

[1] Schulman et al. Gradient Estimation Using Stochastic Computation Graphs. NeurIPS, 2015.

Reviewer_D8ZROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces INFINITYKAN, a Kolmogorov-Arnold network that adapts its number of basis functions during training. A rate parameter controls the basis count and the weights of the active basis functions. The method uses an approximation to a variational objective to train these parameters. Experiments compare the method with fixed-basis KANs and multilayer perceptrons on 18 datasets.

Critical strengths

The method is practical. The appropriate number of basis functions varies across tasks. Automatic selection could reduce the work required to use a KAN.

The experiments are comprehensive. The paper reports results on 18 datasets. INFINITYKAN obtains the highest mean accuracy in 9 datasets in Table 1. This supports the study of adaptive basis selection.

The paper considers total selection cost. It compares adaptive training with a search over fixed basis counts. This comparison is relevant to the proposed practical advantage.

Critical weaknesses

The training objective needs a stronger justification. Equation 9 replaces variational expectations with point evaluations. This approximation does not generally preserve the evidence lower bound, or ELBO. The derivation also drops a term that is constant for each active coefficient. However, its total value changes with the number of active coefficients, so it can affect basis selection.

The stated regularity does not establish stable gradient training. A bound on jumps between integer basis counts does not imply continuity in the real rate parameter. Remark 3.2 acknowledges this distinction, but the broader claims about Lipschitz continuity and stability remain too strong. The method can still be useful as a heuristic, but the current theory does not supply the stated guarantee.

The benefit over simpler capacity selection remains unclear. The approach builds on earlier variational methods for adaptive neural architectures. It introduces prior choices and has about twice the training cost of a fixed model, according to Appendix M. The current results do not clearly establish its advantage over a simple basis-selection method under the same total search budget.

Questions for authors

What is the complete objective when all terms that depend on the active basis count remain in the derivation? Which stability claim remains valid at a basis-count change, rather than only within a fixed-count interval? Does the method keep its advantage when all prior tuning and training costs have the same total selection budget?

1 comment
AuthorsOpenReview ↗

Rebuttal by authors

(W1, Q1) Complete objective with all basis-count-dependent terms.

We agree. Eq. (9) now retains the term arising from evaluating the variational coefficient density at its mean, −ln⁡q(θˉqpkℓKℓ)=12ln⁡(2π)-\ln q(\bar\theta_{qpk}^{\ell K_\ell})=\tfrac{1}{2}\ln(2\pi) per active coefficient, whose aggregate value C(K)=∑ℓnℓ2ln⁡(2π)C(\mathbf K)=\sum_\ell \tfrac{n_\ell}{2}\ln(2\pi), with nℓ=dℓdℓ−1Kℓn_\ell=d_\ell d_{\ell-1}K_\ell, depends on the active basis count. The full derivation, including the gradient ∂C/∂νℓ=−dℓdℓ−12ln⁡(2π) ln⁡(1−τ)/νℓ2\partial C/\partial\nu_\ell = -\tfrac{d_\ell d_{\ell-1}}{2}\ln(2\pi)\,\ln(1-\tau)/\nu_\ell^2 obtained with the relaxed count K~(ν)=−ln⁡(1−τ)/ν\tilde K(\nu)=-\ln(1-\tau)/\nu, is given in App. E.1.

We also agree that evaluating the ELBO integrand at the variational means is an additional approximation that does not in general preserve the lower bound; the revised text calls the objective a first-order ELBO surrogate and bounds the gap by 12LHtr(Σ2)\tfrac12 L_H \mathrm{tr}(\mathbf\Sigma^2) (Prop. E.1).

Because the reviewer correctly notes that this term can in principle affect basis selection, we also measured its effect. We trained a convolutional InfinityKAN on CIFAR10 twice, with C(K)C(K) included and excluded as the only difference between the two runs (Fig. 11, App. M). The learned basis-count trajectories are identical, both converging to a total of 16 bases, and validation accuracy differs by about half a point (80.3 with C(K)C(K) against 80.8 without), the selected configuration reaching 80.88±0.1980.88 \pm 0.19 on test over 10 runs. Including C(K)C(K) therefore shifts the ELBO level without significantly changing the optimization path or the selected architecture. The basis counts stabilize in the first epochs, and the term is nearly constant thereafter.

(W2, Q2) Which stability claim survives at a basis-count change.

We agree that a bound on jumps between integer counts does not imply continuity in the real rate parameter, and we no longer claim it. Theorem 3.1 now states exactly: Kℓ↦ELBO(Kℓ)K_\ell\mapsto\mathrm{ELBO}(K_\ell) is Lipschitz on N\mathbb N; νℓ↦ELBO(Kℓ(νℓ),νℓ)\nu_\ell\mapsto\mathrm{ELBO}(K_\ell(\nu_\ell),\nu_\ell) is piecewise Lipschitz on (0,∞)(0,\infty) with bounded jumps at transitions, hence of special bounded variation on compact intervals. At a transition, the valid quantitative statement is Theorem D.2, which gives an explicit constant for the jump at K↦K+1K\mapsto K+1 in terms of coefficient and basis norms and the network Lipschitz constant; within a fixed-count interval the objective is smooth in νℓ\nu_\ell through the weights wkνℓw_k^{\nu_\ell}. Claims of "stable training dynamics" are now presented as an empirical observation supported by Fig. 4, not as a consequence of the theorem.

(W3, Q3) Advantage under the same total selection budget.

Table 3 is reported end-to-end: for KAN the time includes the sweep over basis-count candidates, for InfinityKAN it includes the sweep over initialization/prior settings plus the per-run resizing overhead.

Under this protocol InfinityKAN is faster end-to-end on 13 of 18 datasets and more accurate on 10 of 18, while evaluating about half as many configurations (App. K). We do not claim a cheaper single run: the ≈2×\approx2\times per-run overhead is stated in the limitations, and it is architecture-dependent rather than intrinsic. To answer the question directly we added App. M.1 (Table 5): a convolutional KAN backbone on CIFAR10 with 8 adaptive layers with identical training protocol, and a model-selection budget matched at 4 configurations per method, where the fixed-basis model sweeps K∈{2,4,8,16}K\in\{2,4,8,16\} and InfinityKAN sweeps only the two priors. InfinityKAN-Conv reaches 87.46±0.3987.46 \pm 0.39 test accuracy against 86.83±0.5186.83 \pm 0.51 for the fixed-basis model (10 runs each), using half as many bases per layer, and the mean per-run wall-clock times are comparable (≈2.9\approx2.9h against ≈3.0\approx3.0h), since the convolutions dominate the cost of resizing. The fixed-basis sweep selected the largest candidate of its grid, so that search was still at its boundary.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal by authors

We thank all reviewers for their careful reading.

We revised the paper to separate clearly (i) the methodological proposal, (ii) the exact objective that is optimized, and (iii) the claims that are formally supported. The three main changes are: the basis-count-dependent normalization term is now retained in Eq. (9) and derived in full (App. E.1); the regularity claim is restated as Lipschitz in the discrete count and piecewise-Lipschitz/SBV in the rate parameter; and the cost discussion is now stated at the level of the end-to-end model-selection budget.

Changed parts are marked in dark blue in the revised document.

AuthorsOpenReview ↗

Summary of the rebuttal and of the revision

We thank the reviewers and the Area Chair. Several comments led to corrections, not only clarifications; changes are highlighted in the updated manuscript.

Replies to the reviews

D8ZR. (i) We agreed that Eq. (9) omitted a count-dependent term: it now keeps 12ln⁡(2π)\tfrac12\ln(2\pi) per active coefficient, derived with its gradient in App. E.1. The objective is now called a first-order ELBO surrogate, with the gap bounded in Prop. E.1. (ii) Bounded jumps do not imply continuity in the rate: Theorem 3.1 now states Lipschitz continuity in the count and piecewise Lipschitz (SBV) in the rate, with the jump constant in Theorem D.2; stability is now an empirical claim. (iii) Cost is argued end-to-end with measured numbers: with about half the configurations, InfinityKAN is faster on 13 of 18 datasets and more accurate on 10 of 18 (Table 3). A convolutional CIFAR-10 backbone at a matched budget of 4 configurations reaches 87.46 (0.39) against 86.83 (0.51), with half the bases per layer and comparable per-run time (App. M.1, Table 6).

eQic. (i) We adopted the reviewer's derivation: the count-only term is proportional to −ln⁡σ2-\ln\sigma^2, so a per-basis penalty arises only for σ>1\sigma>1, and complexity is otherwise controlled by the coefficient magnitudes and the rate prior (Theorem D.3, Remark D.4). Remark E.3 gives the exact coefficient term; an ablation shows the term drives the learned count at equivalent accuracy (App. L, Table 5). (ii) Theorem D.2 now includes the −ln⁡q-\ln q term. (iii) We agreed that an unbounded count (e.g. shifted Poisson) is trainable with score-function estimators. New App. N, acknowledging the suggestion, gives pseudo-code, smoothness of the expected objective (Lemma N.1, Theorem N.2), and memory and variance costs; App. N.1 finds equal accuracy at 2 to 4 times the training time, supporting the deterministic surrogate as default. (iv) AWNN adapts layer dimensions dℓd_\ell; InfinityKAN adapts the basis cardinality KℓK_\ell inside each edge function, which requires KAN-specific results (Theorems A.2, A.3, D.1, D.2). A controlled AWNN comparison is left to future work. (v) App. M.1 states that the selected convolutional configuration keeps its initial count, while the count adapts in the other configurations and in the fully connected runs.

AFAS. (i) App. K now lists what is tuned (the two priors) and what is fixed (τ=0.9\tau=0.9, starting count, weighting distribution, activation). The sensitivity claim is narrowed to the learned counts (Figs. 8, 9); the large-sample argument is in App. O. (ii) The ≈2×\approx2\times per-run overhead is stated as specific to small fully connected models; it is comparable in the convolutional backbone, and we state when a fixed-basis KAN remains preferable. (iii) The 47% on CIFAR-10 comes from a shallow, parameter-matched backbone; the convolutional backbone reaches 87.46%.

Changes to the paper

  • Objective: complete Eq. (9), derivation, gap bound, exact coefficient term (Sec. 3, App. E).
  • Theory: Theorem 3.1 restated; Theorems D.2, D.3 corrected; "Occam penalty linear in KK" removed.
  • Positioning: basis-count against depth or width adaptation; score-function alternative (Related Work, Sec. 3).
  • Cost: end-to-end accounting, tuned against fixed quantities (Table 3, App. K).
  • New experiments: convolutional CIFAR-10 (App. M.1), count-term ablation (App. L), sampled-count comparison (App. N.1).
  • New App. N: score-function variant with regularity results and trade-offs.

Contributions

The number of basis functions per layer is the main open design choice in KANs: it controls capacity, varies across tasks, and multiplies the model-selection grid. InfinityKAN learns it during training. The paper contributes:

  1. A variational KAN with an unbounded number of bases per layer, where a differentiable weighting function links the coefficients and the count distribution, quantile truncation bounds the discarded mass by 1−τ1-\tau, and training uses a deterministic first-order ELBO surrogate with an explicit gap bound.
  2. Theory specific to adapting capacity inside univariate expansions: approximation with increasing bases, soft against hard truncation, the jump at a count transition, and piecewise-Lipschitz regularity in the rate.
  3. Experiments on 18 datasets (synthetic, image, tabular, graph) where InfinityKAN matches or exceeds KANs and MLPs without per-layer basis tuning and at lower selection cost on most datasets, and also works inside a convolutional backbone.

The revision makes the objective explicit about what is approximated, restricts each claim to what is proved or measured, and replaces search-cost arguments with budget-matched comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.