- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes to remove the hyperparameter responsible for selecting the number of basis functions to use in KANs. To do so they propose to use Variational Inference to dynamically learn it via backpropagation. They also solve the zero-gradient problem with soft weighting and develop a specific loss function.
Critical strengths
- I really like the idea of dynamically learning the number of basis functions to use and the tricks used to allow training it. This can eliminate an important KAN bottleneck.
- The paper is well written and easy to follow. The "generative AI tools [used] to assist with LaTeX formatting" were properly employed.
Critical weaknesses
- You are removing one hyperparameter and adding 3/4 of them. Despite you partially state that ("INFINITYKAN introduces two hyperparameters", why didn't you count or the initial basis count?) state that, it seems that you effectively need more tuning. This, added to the fact that each run takes double the time makes me question the effectiveness of the proposed solution.
- The evaluations, despite being abundant, consider only very simple settings. For example you use very shallow networks and achieve only 47% accuracy on CIFAR-10.
Questions for authors
- In Appendix L you try to produce evidence on the fact that some of the added hyperparameters are not important. a) is it the case and based on what evidence? b) Why did you consider only those 2 hyperparameters?
- How does your method extend beyond "trivial" settings / datasets?
3 comments
I thank the authors for the response. I will keep the score
We thank the reviewer for taking the time to review the rebuttal and the modified document.
We would like to ask whether there are still critical issues in our work that did not allow the reviewer to increase the score, and we would appreciate the Reviewer's guidance in this regard.
Thank you again for your efforts in reviewing our work.
Rebuttal by authors
(W1, Qa, Qb) Hyperparameter accounting.
We do not claim zero hyperparameters, and we have replaced the "two hyperparameters" phrasing with explicitly mentioning them. The quantities that are tuned for InfinityKAN are the two priors, η on the rate and σ on the coefficients. The quantile threshold is fixed at τ=0.9 for all 18 datasets and never tuned, as also done in previous works; the starting basis count was varied only on the three synthetic datasets and fixed elsewhere; the weighting distribution and the activation were selected once on the synthetic datasets (truncated exponential, PReLU) and reused unchanged. What is removed is the per-layer basis count, which in a fixed-K KAN multiplies the rest of the grid (m candidates per layer, mL combinations in the worst case) and in our protocol accounted for about half of all configurations (App. K). On the evidence in App. L: Figs. 8 and 9 show, over 18 datasets and 10 seeds, that η and σ mainly control the magnitude and the run-to-run variance of the learned basis counts, with η=10 driving the model to about 2 bases and η=0 leaving counts between 2 and 10. We agree this measures the effect on the learned architecture rather than on accuracy. We have adjusted the claim accordingly; App. N adds the asymptotic argument (Theorem N.1) for why prior influence decreases with sample size [1]. We report the two priors because they are the only new tuned quantities; the others are fixed by protocol, and we now say so explicitly.
[1] Bernardo, J. M. and Smith, A. F. Bayesian theory, volume, 405. John Wiley & Sons, 2009
(W1, continued) Effectiveness given the per-run cost.
We agree the comparison must be made where the cost actually is. The ≈2× per-run overhead is stated in the limitations, and it is specific to the small fully-connected setting, where resizing the coefficient tensors is a significant fraction of the step cost. In the convolutional setting added during the rebuttal, the per-run wall-clock of InfinityCKAN and CKAN is comparable because the convolutions dominate. At the pipeline level, Table 3 reports end-to-end times including model selection for both methods: InfinityKAN is faster on 13 of 18 datasets and more accurate on 10 of 18, with about half the configurations. We have revised the Model selection paragraph and the Table 3 caption to state this trade-off explicitly, including the regime where fixed-basis KAN remains preferable.
(W2, Qc) CIFAR-10 accuracy and beyond simple settings.
The 47% in Table 1 comes from a deliberately shallow, parameter-matched backbone shared by KAN, MLP and InfinityKAN, chosen for a controlled comparison rather than for absolute accuracy; within that setting InfinityKAN gives the largest relative gains on the hardest datasets (CIFAR10 43.00 to 46.99, CIFAR100 14.04 to 19.27). To address the concern directly, we added App. M.1 (Table 5), where all 8 layers of a convolutional KAN backbone adapt their basis count: InfinityKAN-Conv reaches 87.46±0.39 test accuracy against 86.83±0.51 for the fixed-basis counterpart whose basis count was selected by sweep, over 10 runs each, under an identical protocol and a budget matched at 4 configurations per method. A second and smaller convolutional backbone is the one behind the C(K) study of Fig. 11, at 80.88±0.19; its fixed-basis counterpart is still training, so we label the convolutional results as partial and will complete them in the revision. Already at this stage, they show that the 47% follows from the shallow backbone rather than from the adaptive mechanism, and that basis adaptation keeps working when the KAN layers sit inside a convolutional architecture.