- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies memorization in Kolmogorov-Arnold Networks (KANs) under corrupted labels. The authors decompose the population-level prediction change caused by a corrupted label into two components: the corruption fitting, , and the population footprint of the resulting response, , with
Under a localized, frozen-NTK analysis of two-layer B-spline KANs, the authors analyze how grid size and spline degree affect corruption fitting and its population footprint. Refining the grid accelerates corruption fitting while making the response more localized; increasing the spline degree slows fitting while broadening the response. Under the stated assumptions, the authors obtain
This analysis explains why comparisons at a fixed training budget may differ from comparisons at a matched level of corruption fitting. Building on this analysis, the authors propose Curvature-Guided Basis Perturbations (CURB), which perturbs basis functions along normalized curvature directions during training and restores the original basis functions at inference. In a local frozen-Jacobian MSE model, the authors prove a short-time reduction in corruption fitting. Under an additional commutativity condition, they also establish an ordering over the entire local trajectory. Experiments cover synthetic regression, California Housing, Darcy operator learning, MNIST with corrupted labels, USPS transfer, and multiple basis families. The results show that CURB reduces corruption fitting and improves clean-data prediction in several settings. In the reported MNIST setting, it achieves accuracy close to SAM at a lower per-update cost.
Critical strengths
-
The paper separates how much of a corrupted label is fitted from how broadly the resulting response spreads through the input population. This reveals two effects of grid refinement: faster corruption fitting and a more localized response. The distinction helps explain why fixed-time and matched-fit comparisons can yield different conclusions. These comparisons are an important part of the paper’s contribution and may also improve the design and interpretation of future robustness experiments for KANs.
-
CURB acts directly on the basis functions that define KAN edges. It does not require prior knowledge of which samples are corrupted, and it does not introduce perturbations at inference time. The experiments include regression, classification, scientific operator learning, and multiple basis families. In the fixed-coefficient-budget MNIST comparison, CURB achieves an average accuracy close to SAM while requiring less computation per update. These results support its practical value.
Critical weaknesses
-
The scope of the connection between the theoretical results and the practical experiments needs clearer boundaries. The scaling laws in Section 4 are derived under specific conditions, including two-layer B-spline KANs, localization, a frozen NTK, and gradient-flow training. The experiments also involve finite-width networks, AdamW optimization, and three-layer KANs. Observing trends consistent with the theory provides useful support, but does not establish that the same scaling relationships hold quantitatively in these practical settings. For Theorem 6, the short-time reduction in corruption fitting should also be distinguished from the ordering over the entire local trajectory, which requires an additional commutativity condition. If the local-response probes start from checkpoints independently trained by ERM and CURB, the observed differences may reflect both the perturbations applied during the probes and representation differences induced by prior training. The paper should more clearly identify which experimental results the theory explains and which are empirical observations beyond its assumptions.
-
The independent contribution of curvature-guided directions and their selective effect on corruption fitting have not been sufficiently established. The short-time reduction in Theorem 6 follows from a positive-semidefinite second-order term induced by a zero-mean Jacobian perturbation. This analysis does not by itself establish that curvature directions are more effective than other basis-function perturbation directions, nor does it guarantee that CURB preferentially suppresses corrupted-label fitting while preserving the learning of genuine signal. Comparisons with SAM, ASAM, input noise, and DropKAN help evaluate CURB’s overall effectiveness, but do not directly isolate the contribution of the curvature-guided design. A comparison with random directions under matched perturbation norms would also help explain CURB’s differing performance across tasks.
-
The theoretical connection between reduced corruption fitting and reduced population-level malign memorization remains incomplete. The paper defines malign memorization as , whereas Theorem 6 analyzes the corruption-fitting term without characterizing how CURB affects the population footprint . Consequently, reducing alone does not guarantee a reduction in if increases sufficiently. The reported reductions in corruption fitting and improvements in clean-test error are encouraging, but do not directly establish that CURB reduces the paper’s defined population-level memorization measure .
Questions for authors
-
Starting points and causal attribution in the local-response experiment. Please clarify whether the local corruption-response probes for ERM and CURB start from the same parameter checkpoint. If they start from checkpoints trained separately by each method, does the comparison also reflect representation differences induced by prior training? Is there an existing comparison that starts from the same checkpoint and changes only whether CURB is enabled, to distinguish the direct effect of perturbation from model changes induced by the preceding training process? If the experiment is intended to evaluate the combined effect of the two complete training procedures, please make this explicit.
-
The relationship between Theorem 6’s conditions and the experimental conclusions. Please distinguish which experimental results primarily support the short-time reduction in Theorem 6 and which are used to support an interpretation of reduced corruption fitting over the entire local trajectory. Is the commutativity condition required for the latter treated as an idealized sufficient condition, or do the authors believe it is approximately satisfied in the practical experiments? If this condition has been measured or analyzed, please report or point to the relevant evidence. Otherwise, please further qualify the scope of the theory’s interpretation of the practical training trajectories.
-
The independent contribution of curvature-guided directions. What evidence distinguishes the effect of curvature-guided perturbations from that of general random basis-function perturbations? Are there random-direction controls with matched perturbation norms, sampling procedures, and training budgets? Additionally, has CURB’s effect on responses to clean labels and corrupted labels been compared, to assess whether it preferentially suppresses corruption fitting while preserving the learning of genuine signal?
-
The effect of CURB on and . How does the population footprint change under CURB? Can the authors report , , and their product for the same corrupted sample, or otherwise show that the decrease in is not offset by an increase in ? This would help determine whether CURB reduces the paper’s defined population-level malign memorization measure, in addition to slowing corruption fitting.
-
Statistical robustness and comparability of hyperparameter selection. The main Darcy comparison and the fixed-coefficient-budget MNIST table use a limited number of repetitions. Please provide existing multi-seed results or further clarify the stability of these comparisons. Please also clarify whether CURB, SAM, and ASAM use comparable hyperparameter search ranges and consistent validation-set selection procedures.
2 comments
Author rebuttal (2/2)
W3,Q4) CURB effectively reduces during training; offsets
Under the assumptions of Theorem 6, our analysis characterizes CURB's effect on corruption fitting . We complement this result by directly measuring and to determine the resulting population-level benefit. Hence, we report , , and against training time with fixed using five seeds (Appendix C.1). Complementary grid size sweeps use both fixed strengths and , also with five seeds (Figure 8).
In Fig. 8, at approximately 5,000 updates, both strengths reduce mean at every grid size. For , the reduction holds in all 50 paired comparisons. The results agree with the reviewer's expectations; can increase, yet it is offset by , demonstrating reduced population-level malign memorization for CURB-trained models.
Q5a) Extended Darcy comparison and MNIST comparison retain the trends in Table 2 with five seeds
On Darcy, CURB achieves relative error versus SAM’s , with lower error in every paird seed. We also included ASAM, which attained relative error. On MNIST, CURB achieves accuracy and improves over ERM in every evaluated configuration. The MNIST extension cover all six methods and seven configurations, totaling 210 runs, with the original experimental settings and training budgets retained.
Q5b) Hyperparameter selection
Appendix C documents the hyperparameter selection and experimental settings. The original smooth-regression CURB search was capped at ; extending it to selects by independent clean validation. CURB then attains test MSE , making it competitive with SAM and ASAM. We report this in-text while retaining the original smooth-regression row in Table 1 for consistency with the original HPO ranges.
Kind regards,
Authors
Author rebuttal (1/2)
Dear Reviewer sUew,
We are thankful for your careful analysis and feedback along with finding our work to be informative and well-presented, and finding the proposed solution to be of practical value.
We apologize for the late response.
Below we address the stated weaknesses and questions. In the revised PDF, we tagged the relevant parts with same enumeration and orange, bold text in square brackets.
Q1) Common starting points separate CURB's direct effect from prior representation differences
The original local-response probes start from each method's own trained checkpoint and characterize the learned solutions. We additionally include controlled comparisons in which ERM and CURB begin from identical parameters: either the same clean-trained checkpoint or the same random initialization (Appendix C.1). Within each comparison, training data and sample presentation order are matched, and only CURB is changed.
Both starting conditions receive 10K post-start GD updates over five paired seeds. The clean-trained starts receive 4K clean AdamW updates beforehand, used to stabilize the local-response measurement; random starts receive no pretraining. CURB reduces corruption-induced population response from both starting conditions, showing that the observed benefit also occurs without clean pretraining.
W1,Q2) Short-time suppression is guaranteed; sustained suppression is empirically supported.
As stated in Theorem 6, the short-time suppression result does not require commutativity, whereas the full-trajectory ordering in Eq. (19) does. We treat commutativity as an idealized sufficient condition; we do not assume approximate commutativity when interpreting practical training. The longer-time behavior is now supported empirically by the five-seed comparisons in Appendix C.1, Fig. 15, from both shared clean-trained checkpoints and random initialization.
Section 6 now distinguishes the theoretical setting from the broader empirical benchmarks. Appendix C.3 (Figure 16) additionally checks the frozen-tangent approximation against nonlinear GD and AdamW training.
W2,Q3a) The independent contribution of curvature-guided directions
To isolate the independent contribution of curvature-guided directions, we include controls with random basis-function perturbations and first- and third-derivative directions, using matched expected perturbation energy and training budgets (Appendix C.2, Table 5). Across seven matched-capacity MNIST configurations and three paired seeds, CURB achieves 89.34±0.30% accuracy, versus 87.00±1.61% for random perturbations, 88.37±0.47% for first derivatives, and 87.80±0.43% for third derivatives. Third derivatives vanish for p≤2, whereas curvature perturbations remain active for quadratic splines.
For a more direct investigation of causality (Fig.11) between CURB’s suppression of corruption fitting and its generalization benefit, we also designed an intervention-based experiment where we explicitly enforced corruption fitting while keeping CURB active, increasing the fraction of corruption fitted from 61.4% to 97.5%. This eliminated CURB’s clean-test advantage across all three paired seeds, providing evidence for the causal link between reduced corruption fitting and improved test generalization.
Q3b) CURB preserves genuine-signal learning while suppressing corruption fitting
We include a dedicated paired comparison of clean-signal learning and corrupted-label fitting. ERM and CURB train from the same random initialization on both clean and corrupted targets, with identical minibatches and the same 4,800-update budget. The smooth 2D regression task uses 20% corrupted labels, G=16,32, and three paired seeds; corruption fitting is measured relative to each method's clean-label twin.
At G=16, CURB removes 99.82±0.04% of its initial clean-test error while fitting 24.53±0.35% of the injected corruption; at G=32, it removes 99.94±0.02% while fitting 61.43±1.01%. ERM fits essentially all of the corruption. Over the full training trajectory, CURB retains approximately all of ERM's signal-learning area but only 23.34±0.60% and 55.78±0.33% of its corruption-fitting area at the two grids. The signal-versus-corruption retention contrast holds in all six paired seed/grid comparisons. These results directly support preferential suppression of corruption while preserving genuine-signal learning in the tested setting.