acceptodds
Under review as a conference paper at ICLR 2027

Plottability Is Not Identifiability: Auditing the Retraining Stability of KAN Explanations

Abstract

Kolmogorov–Arnold Networks (KANs) are promoted as interpretable because every edge is a plottable univariate function. We audit that claim by asking whether the plots survive retraining, and find that on tabular data the stability credited to KANs tracks additivity; we detect no contribution from the edge architecture. A plotted shape is worth reading only if retraining an equally accurate model reproduces it. We measure this retraining stability, a lower bound on disagreement over the Rashomon set, under a leakage-free protocol across 14 tabular datasets, five splits, three shape extractors (PDP, ALE, SHAP) and three bases. Spline and boosted additive models reproduce their plots; a deep KAN's do not, on 9/9 regression and 5/5 classification datasets, the gap clearing SE on 13/14 and never inverting at a common held-out loss radius. A deep KAN is no more stable than a deep MLP (7/14); an apparent KAN edge at high capacity coincides with the MLP carrying more parameters and is not detected near parity, against an additivity effect of –. No deep model of B-spline, RBF or Chebyshev basis approaches the spline additive reference. A matmul-only profiler misses – of a deep KAN's operations; counting them, its arithmetic is – the MLP's. We then draw the boundary of our own standard: the hierarchy inverts where an additive model is misspecified, the accuracy gap is often not about interactions, and controlling knot placement leaves the direction intact on 14/14 while halving a magnitude we decline to claim. The ordering holds from k to k rows. Finally we turn the standard on itself: four of five interaction screens reproduce their own top pairs across seeds (median Jaccard , the fifth failing it) while agreeing with one another at , and KAN's own prune-and-symbolify workflow—on Feynman equations, where a formula exists to find—does not converge on a structure across seeds and recovers the generating equation on at most one seed in five, while every seed's formula predicts alike. Cross-seed stability is necessary but not sufficient; a method can be perfectly reproducible and still arbitrary.

Primary area:
interpretability and explainable AI
Keywords:
Kolmogorov-Arnold networks, interpretability, identifiability, generalized additive models, Rashomon set, partial dependence, symbolic regression, reproducibility

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_jBNiOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Kolmogorov Arnold Networks (KANs) are a (somewhat) new architecture which offers interpretability gains because edge functions can be plotted, so eyeballing them can give some explanation on what is happening (similar to how CNN masks used to be looked at). The authors ask a number of questions: is this stable under retraining, what is stable under KAN retraining, how does this compare to MLP stability, and what would be more stable. However the main point the authors make is that since they find that the plots are unstable under training, this undermines the interpertability benefits claimed by KANs, which they quantify by testing a disagreement method across different training runs on various datasets. Based on their findings they suggest a GrowKAN model (somewhat buried in the paper).

Critical strengths
  1. The main contribution is a point which is at once somewhat common-sense but can also be overlooked - when a KAN is trained, and some pleasing figures are formed, there's a question of how stable they are. I think this could sometimes be ignored in practice, and is genuinely worth point out.
  2. The execution of experiments is very good, with robustness covering many aspects (SHAP, PDP) and spline choices. The use of 14 datasets is good (with good coverage).
  3. The reporting is very good, with numerous low-points conceded, reproducibility details given.
Critical weaknesses

I unfortunately consider the paper weak due to a single main point, with a large number of manifestations. It basically all boils down to the main abstract point "A plotted shape is worth reading only if retraining an equally accurate model reproduces it." This statement could be correct or not, its correctness probably depends on new definitions that would need to be developed way beyond the paper (and are not necessarily KAN related), and, in any case, this specific paper works on many aspects of related questions, but does not convince that this claim has been addressed, let alone studied.

  1. Conceptually: a. There can be many useful explanations, each shedding light on some facet, either in shallow representation, or in deeper ways. To start with the most extreme example, if a tabular dataset contains two columns: income by USD, income by Euro, then virtually the same explanation could boil down to something giving the insight from either of the two columns, which the paper's methodology would flag as failing. At the other extreme, any new study on another carcinogenic factor shows an example of useful distinct explanations which are not fundamentally the same, and the question of whether KAN instability would cycle between these or not is interesting, but the paper doesn't really advance insights about it. To clarify the two extremes, in the first case the user could say, the model chose to highlight each time the same explanation via focussing on a different column, in the second case the user could say that there were 5 explanations, each of which is useful; whether this happens or not IDK, but after reading the paper I don't see more whether KANs do this or not. To the credit of the authors they did somewhat list human relevance Conclusions, but I think it should be in Limitations, and it's a fundamental limitation on the paper.
  2. Even when pinpointing things, there's a gap between the declarations and the execution. The formal quantity considered is max difference among all models in a class with loss eps from optimal. The experiments measure among a few models trained with different seeds settings, average explanation difference. This is either not the same, or not nearly motivated enough to explain the same. \hat D_j depends on the optimizer-induced distribution over solutions, whereas D_j is a worst-case property of the full Rashomon set. Therefore, comparisons of \hat D_j may compare optimization behavior rather than intrinsic identifiability of model classes.

Therefore, I unfortunately feel that the paper has a number of severe disconnects from a series of experiments to a far reaching claim.

Questions for authors
  1. Conceptually, why does an explanation of one specific fitted model need to reproduce across other equally accurate models in order to be worth reading? How do you distinguish equivalent, complementary, genuinely contradictory, and useless explanations?

  2. Technically, since \hat D_j depends on the small set of solutions reached by a particular optimizer, what justifies using it to draw conclusions about D_j, which is a worst-case property of the full Rashomon set?

  3. Could the results support only the narrower claim that KAN plots should not be treated as model-independent properties of the data, rather than the broader claim that unstable plots are not interpretable or worth reading?

Reviewer_E37bOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the shape plots of KANs are reproduced when an equally accurate model is retrained with a different seed. Across 14 tabular datasets, the authors find that additive models are more stable than deep KANs, and that deep KANs are no more stable than deep MLPs. They conclude that stability comes from additivity rather than the KAN architecture, and propose reporting cross-seed shape disagreement alongside accuracy.

Critical strengths

The experiments are extensive and carefully controlled, the limitations are reported candidly, and the reproducibility infrastructure is strong.

Critical weaknesses
  1. Limited novelty.

Limited novelty. That additive models yield stable shapes while non-additive models do not is already well known, as the authors acknowledge, and the similarity between deep KANs and MLPs is also expected. The paper mainly confirms known results rather than offering new insight.

  1. Weak theoretical contribution.

Proposition 1 mostly restates known results: part (i) follows directly from strong convexity and the triangle inequality, and parts (ii) and (iii) are constructed examples. As the authors note, it does not apply to trained networks or to classification, and the proposed concurvity mechanism is not observed on real data.

  1. Unclear practical value of the proposed criterion.

Under the authors' own criterion, EBM turns out to be the best model. GrowKAN, which was designed specifically to meet this criterion, is Pareto non-dominated on only 3/14 datasets and is dominated by EBM-GA2M. If an existing model is best under the criterion, does it have practical value in guiding the design of new models, or is it only a post-hoc reporting tool?

  1. Missing related work on identifiability.

Since identifiability is the central topic of this paper, a much more thorough literature review on this topic is needed. In particular, ANOVA-TPNN ([1]), a representative work on identifiability in additive neural networks, is not discussed. The authors should position their contribution relative to such work and ideally include it as a baseline.

[1]. Park, Seokhun, et al. "Tensor product neural networks for functional anova model." arXiv preprint arXiv:2502.15215 (2025).

Questions for authors
  1. Can you give a concrete case where the proposed criterion leads to a modeling decision that existing practice would not?

  2. How does the notion of identifiability proposed in this paper differ from identifiability in the functional ANOVA framework?

Reviewer_Zv2iOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the interpretability of Kolmogorov Arnold Networks (KANs) by evaluating the stability of their explanations across different random seed retrainings. The authors propose a cross seed disagreement metric to assess explanation identifiability and conduct extensive experiments on tabular datasets, comparing KANs with additive models and MLPs. The results suggest that explanation stability is primarily attributed to additive structures rather than the KAN architecture itself. The paper further examines the stability of interaction discovery and symbolic expression extraction, highlighting the limitations of equating plottability with interpretability.

Critical strengths
  1. The paper conducts extensive experiments across multiple datasets and KAN architectures.
Critical weaknesses
  1. Limited Methodological Novelty. The paper primarily presents an empirical evaluation of explanation stability across existing models rather than introducing a novel algorithm or methodology. It does not propose a new learning algorithm or optimization strategy to improve interpretability. Overall, the work reads more like an extensive experimental report on KAN interpretability than a study offering substantial methodological innovation.

  2. The paper defines identifiability using the maximum explanation disagreement over a Rashomon set but approximates it with average pairwise disagreement across a few retrained models. This metric is sensitive to initialization and optimization choices and may fail to capture the full diversity of near-optimal solutions. Therefore, low cross-seed disagreement does not necessarily imply global identifiability.

  3. The experiments are primarily conducted on low-dimensional tabular datasets with moderate sample sizes. Although synthetic and Feynman datasets are included, the conclusions remain insufficiently validated on more challenging settings, such as high-dimensional data, larger KAN architectures.

Questions for authors

See the weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.