- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Kolmogorov Arnold Networks (KANs) are a (somewhat) new architecture which offers interpretability gains because edge functions can be plotted, so eyeballing them can give some explanation on what is happening (similar to how CNN masks used to be looked at). The authors ask a number of questions: is this stable under retraining, what is stable under KAN retraining, how does this compare to MLP stability, and what would be more stable. However the main point the authors make is that since they find that the plots are unstable under training, this undermines the interpertability benefits claimed by KANs, which they quantify by testing a disagreement method across different training runs on various datasets. Based on their findings they suggest a GrowKAN model (somewhat buried in the paper).
Critical strengths
- The main contribution is a point which is at once somewhat common-sense but can also be overlooked - when a KAN is trained, and some pleasing figures are formed, there's a question of how stable they are. I think this could sometimes be ignored in practice, and is genuinely worth point out.
- The execution of experiments is very good, with robustness covering many aspects (SHAP, PDP) and spline choices. The use of 14 datasets is good (with good coverage).
- The reporting is very good, with numerous low-points conceded, reproducibility details given.
Critical weaknesses
I unfortunately consider the paper weak due to a single main point, with a large number of manifestations. It basically all boils down to the main abstract point "A plotted shape is worth reading only if retraining an equally accurate model reproduces it." This statement could be correct or not, its correctness probably depends on new definitions that would need to be developed way beyond the paper (and are not necessarily KAN related), and, in any case, this specific paper works on many aspects of related questions, but does not convince that this claim has been addressed, let alone studied.
- Conceptually: a. There can be many useful explanations, each shedding light on some facet, either in shallow representation, or in deeper ways. To start with the most extreme example, if a tabular dataset contains two columns: income by USD, income by Euro, then virtually the same explanation could boil down to something giving the insight from either of the two columns, which the paper's methodology would flag as failing. At the other extreme, any new study on another carcinogenic factor shows an example of useful distinct explanations which are not fundamentally the same, and the question of whether KAN instability would cycle between these or not is interesting, but the paper doesn't really advance insights about it. To clarify the two extremes, in the first case the user could say, the model chose to highlight each time the same explanation via focussing on a different column, in the second case the user could say that there were 5 explanations, each of which is useful; whether this happens or not IDK, but after reading the paper I don't see more whether KANs do this or not. To the credit of the authors they did somewhat list human relevance Conclusions, but I think it should be in Limitations, and it's a fundamental limitation on the paper.
- Even when pinpointing things, there's a gap between the declarations and the execution. The formal quantity considered is max difference among all models in a class with loss eps from optimal. The experiments measure among a few models trained with different seeds settings, average explanation difference. This is either not the same, or not nearly motivated enough to explain the same. \hat D_j depends on the optimizer-induced distribution over solutions, whereas D_j is a worst-case property of the full Rashomon set. Therefore, comparisons of \hat D_j may compare optimization behavior rather than intrinsic identifiability of model classes.
Therefore, I unfortunately feel that the paper has a number of severe disconnects from a series of experiments to a far reaching claim.
Questions for authors
-
Conceptually, why does an explanation of one specific fitted model need to reproduce across other equally accurate models in order to be worth reading? How do you distinguish equivalent, complementary, genuinely contradictory, and useless explanations?
-
Technically, since \hat D_j depends on the small set of solutions reached by a particular optimizer, what justifies using it to draw conclusions about D_j, which is a worst-case property of the full Rashomon set?
-
Could the results support only the narrower claim that KAN plots should not be treated as model-independent properties of the data, rather than the broader claim that unstable plots are not interpretable or worth reading?