acceptodds
Under review as a conference paper at ICLR 2027

Scaling Laws and Mitigation of Memorization in Kolmogorov Arnold Networks

Abstract

Malign memorization limits the deployment of Kolmogorov-Arnold Networks (KANs) under corrupted labels, particularly in scientific machine learning where observational measurements contain label corruptions. While empirical studies observe this performance degradation, existing work does not explain how the defining KAN controls, grid size and spline degree , govern the resulting population damage. We analyze this behavior by decomposing memorization into corruption fitting and its population footprint , giving . In the localized regime of a two-layer B-spline KAN, grid refinement accelerates corruption fitting while contracting its footprint, whereas increasing degree slows fitting and broadens the footprint, yielding the matched-fit scaling . This decomposition explains why fixed-budget training often report worse memorization at larger , even though matched-fit memorization becomes strictly more localized. Guided by this analysis, we propose Curvature-Guided Basis Perturbations (CURB), which provably slows corruption fitting by perturbing curvature-sensitive basis directions during training. Across function regression, scientific operator learning (Darcy flow), and classification benchmarks, CURB reduces label corruption fitting, achieves competitive or superior accuracy to SAM and ASAM at lower per-update cost, and transfers successfully across diverse KAN basis families. In addition, CURB consistently leads trained models to flatter empirical loss landscapes.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
malign memorization, Kolmogorov Arnold Networks, generalization, flat-minima

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_sUewOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies memorization in Kolmogorov-Arnold Networks (KANs) under corrupted labels. The authors decompose the population-level prediction change caused by a corrupted label into two components: the corruption fitting, αc(t)\alpha_c(t), and the population footprint of the resulting response, Vc(t)V_c(t), with

Mc(t)=αc(t)2Vc(t).M_c(t)=\alpha_c(t)^2V_c(t).

Under a localized, frozen-NTK analysis of two-layer B-spline KANs, the authors analyze how grid size GG and spline degree pp affect corruption fitting and its population footprint. Refining the grid accelerates corruption fitting while making the response more localized; increasing the spline degree slows fitting while broadening the response. Under the stated assumptions, the authors obtain

Mc(t)=Θ ⁣(αc(t)2pG).M_c(t)=\Theta\!\left(\alpha_c(t)^2\frac{\sqrt p}{G}\right).

This analysis explains why comparisons at a fixed training budget may differ from comparisons at a matched level of corruption fitting. Building on this analysis, the authors propose Curvature-Guided Basis Perturbations (CURB), which perturbs basis functions along normalized curvature directions during training and restores the original basis functions at inference. In a local frozen-Jacobian MSE model, the authors prove a short-time reduction in corruption fitting. Under an additional commutativity condition, they also establish an ordering over the entire local trajectory. Experiments cover synthetic regression, California Housing, Darcy operator learning, MNIST with corrupted labels, USPS transfer, and multiple basis families. The results show that CURB reduces corruption fitting and improves clean-data prediction in several settings. In the reported MNIST setting, it achieves accuracy close to SAM at a lower per-update cost.

Critical strengths
  1. The paper separates how much of a corrupted label is fitted from how broadly the resulting response spreads through the input population. This reveals two effects of grid refinement: faster corruption fitting and a more localized response. The distinction helps explain why fixed-time and matched-fit comparisons can yield different conclusions. These comparisons are an important part of the paper’s contribution and may also improve the design and interpretation of future robustness experiments for KANs.

  2. CURB acts directly on the basis functions that define KAN edges. It does not require prior knowledge of which samples are corrupted, and it does not introduce perturbations at inference time. The experiments include regression, classification, scientific operator learning, and multiple basis families. In the fixed-coefficient-budget MNIST comparison, CURB achieves an average accuracy close to SAM while requiring less computation per update. These results support its practical value.

Critical weaknesses
  1. The scope of the connection between the theoretical results and the practical experiments needs clearer boundaries. The scaling laws in Section 4 are derived under specific conditions, including two-layer B-spline KANs, localization, a frozen NTK, and gradient-flow training. The experiments also involve finite-width networks, AdamW optimization, and three-layer KANs. Observing trends consistent with the theory provides useful support, but does not establish that the same scaling relationships hold quantitatively in these practical settings. For Theorem 6, the short-time reduction in corruption fitting should also be distinguished from the ordering over the entire local trajectory, which requires an additional commutativity condition. If the local-response probes start from checkpoints independently trained by ERM and CURB, the observed differences may reflect both the perturbations applied during the probes and representation differences induced by prior training. The paper should more clearly identify which experimental results the theory explains and which are empirical observations beyond its assumptions.

  2. The independent contribution of curvature-guided directions and their selective effect on corruption fitting have not been sufficiently established. The short-time reduction in Theorem 6 follows from a positive-semidefinite second-order term induced by a zero-mean Jacobian perturbation. This analysis does not by itself establish that curvature directions are more effective than other basis-function perturbation directions, nor does it guarantee that CURB preferentially suppresses corrupted-label fitting while preserving the learning of genuine signal. Comparisons with SAM, ASAM, input noise, and DropKAN help evaluate CURB’s overall effectiveness, but do not directly isolate the contribution of the curvature-guided design. A comparison with random directions under matched perturbation norms would also help explain CURB’s differing performance across tasks.

  3. The theoretical connection between reduced corruption fitting and reduced population-level malign memorization remains incomplete. The paper defines malign memorization as Mc(t)=αc(t)2Vc(t)M_c(t)=\alpha_c(t)^2V_c(t), whereas Theorem 6 analyzes the corruption-fitting term αc(t)\alpha_c(t) without characterizing how CURB affects the population footprint Vc(t)V_c(t). Consequently, reducing αc(t)\alpha_c(t) alone does not guarantee a reduction in Mc(t)M_c(t) if Vc(t)V_c(t) increases sufficiently. The reported reductions in corruption fitting and improvements in clean-test error are encouraging, but do not directly establish that CURB reduces the paper’s defined population-level memorization measure Mc(t)M_c(t).

Questions for authors
  1. Starting points and causal attribution in the local-response experiment. Please clarify whether the local corruption-response probes for ERM and CURB start from the same parameter checkpoint. If they start from checkpoints trained separately by each method, does the comparison also reflect representation differences induced by prior training? Is there an existing comparison that starts from the same checkpoint and changes only whether CURB is enabled, to distinguish the direct effect of perturbation from model changes induced by the preceding training process? If the experiment is intended to evaluate the combined effect of the two complete training procedures, please make this explicit.

  2. The relationship between Theorem 6’s conditions and the experimental conclusions. Please distinguish which experimental results primarily support the short-time reduction in Theorem 6 and which are used to support an interpretation of reduced corruption fitting over the entire local trajectory. Is the commutativity condition required for the latter treated as an idealized sufficient condition, or do the authors believe it is approximately satisfied in the practical experiments? If this condition has been measured or analyzed, please report or point to the relevant evidence. Otherwise, please further qualify the scope of the theory’s interpretation of the practical training trajectories.

  3. The independent contribution of curvature-guided directions. What evidence distinguishes the effect of curvature-guided perturbations from that of general random basis-function perturbations? Are there random-direction controls with matched perturbation norms, sampling procedures, and training budgets? Additionally, has CURB’s effect on responses to clean labels and corrupted labels been compared, to assess whether it preferentially suppresses corruption fitting while preserving the learning of genuine signal?

  4. The effect of CURB on Vc(t)V_c(t) and Mc(t)M_c(t). How does the population footprint Vc(t)V_c(t) change under CURB? Can the authors report αc(t)\alpha_c(t), Vc(t)V_c(t), and their product Mc(t)M_c(t) for the same corrupted sample, or otherwise show that the decrease in αc(t)2\alpha_c(t)^2 is not offset by an increase in Vc(t)V_c(t)? This would help determine whether CURB reduces the paper’s defined population-level malign memorization measure, in addition to slowing corruption fitting.

  5. Statistical robustness and comparability of hyperparameter selection. The main Darcy comparison and the fixed-coefficient-budget MNIST table use a limited number of repetitions. Please provide existing multi-seed results or further clarify the stability of these comparisons. Please also clarify whether CURB, SAM, and ASAM use comparable hyperparameter search ranges and consistent validation-set selection procedures.

2 comments
AuthorsOpenReview ↗

Author rebuttal (1/2)

Dear Reviewer sUew,

We are thankful for your careful analysis and feedback along with finding our work to be informative and well-presented, and finding the proposed solution to be of practical value.
We apologize for the late response.

Below we address the stated weaknesses and questions. In the revised PDF, we tagged the relevant parts with same enumeration and orange, bold text in square brackets.

Q1) Common starting points separate CURB's direct effect from prior representation differences

The original local-response probes start from each method's own trained checkpoint and characterize the learned solutions. We additionally include controlled comparisons in which ERM and CURB begin from identical parameters: either the same clean-trained checkpoint or the same random initialization (Appendix C.1). Within each comparison, training data and sample presentation order are matched, and only CURB is changed.

Both starting conditions receive 10K post-start GD updates over five paired seeds. The clean-trained starts receive 4K clean AdamW updates beforehand, used to stabilize the local-response measurement; random starts receive no pretraining. CURB reduces corruption-induced population response from both starting conditions, showing that the observed benefit also occurs without clean pretraining.

W1,Q2) Short-time suppression is guaranteed; sustained suppression is empirically supported.

As stated in Theorem 6, the short-time suppression result does not require commutativity, whereas the full-trajectory ordering in Eq. (19) does. We treat commutativity as an idealized sufficient condition; we do not assume approximate commutativity when interpreting practical training. The longer-time behavior is now supported empirically by the five-seed comparisons in Appendix C.1, Fig. 15, from both shared clean-trained checkpoints and random initialization.

Section 6 now distinguishes the theoretical setting from the broader empirical benchmarks. Appendix C.3 (Figure 16) additionally checks the frozen-tangent approximation against nonlinear GD and AdamW training.

W2,Q3a) The independent contribution of curvature-guided directions

To isolate the independent contribution of curvature-guided directions, we include controls with random basis-function perturbations and first- and third-derivative directions, using matched expected perturbation energy and training budgets (Appendix C.2, Table 5). Across seven matched-capacity MNIST configurations and three paired seeds, CURB achieves 89.34±0.30%89.34\pm0.30\% accuracy, versus 87.00±1.61%87.00\pm1.61\% for random perturbations, 88.37±0.47%88.37\pm0.47\% for first derivatives, and 87.80±0.43%87.80\pm0.43\% for third derivatives. Third derivatives vanish for p≤2p\le2, whereas curvature perturbations remain active for quadratic splines.

For a more direct investigation of causality (Fig.11) between CURB’s suppression of corruption fitting and its generalization benefit, we also designed an intervention-based experiment where we explicitly enforced corruption fitting while keeping CURB active, increasing the fraction of corruption fitted from 61.4% to 97.5%. This eliminated CURB’s clean-test advantage across all three paired seeds, providing evidence for the causal link between reduced corruption fitting and improved test generalization.

Q3b) CURB preserves genuine-signal learning while suppressing corruption fitting

We include a dedicated paired comparison of clean-signal learning and corrupted-label fitting. ERM and CURB train from the same random initialization on both clean and corrupted targets, with identical minibatches and the same 4,800-update budget. The smooth 2D regression task uses 20% corrupted labels, G=16,32G=16,32, and three paired seeds; corruption fitting is measured relative to each method's clean-label twin.

At G=16G=16, CURB removes 99.82±0.04%99.82\pm0.04\% of its initial clean-test error while fitting 24.53±0.35%24.53\pm0.35\% of the injected corruption; at G=32G=32, it removes 99.94±0.02%99.94\pm0.02\% while fitting 61.43±1.01%61.43\pm1.01\%. ERM fits essentially all of the corruption. Over the full training trajectory, CURB retains approximately all of ERM's signal-learning area but only 23.34±0.60%23.34\pm0.60\% and 55.78±0.33%55.78\pm0.33\% of its corruption-fitting area at the two grids. The signal-versus-corruption retention contrast holds in all six paired seed/grid comparisons. These results directly support preferential suppression of corruption while preserving genuine-signal learning in the tested setting.

AuthorsOpenReview ↗

Author rebuttal (2/2)

W3,Q4) CURB effectively reduces Mc(t)M_c(t) during training; αc(t)2\alpha_c(t)^2 offsets Vc(t)V_c(t)

Under the assumptions of Theorem 6, our analysis characterizes CURB's effect on corruption fitting αc(t)\alpha_c(t). We complement this result by directly measuring Vc(t)V_c(t) and Mc(t)M_c(t) to determine the resulting population-level benefit. Hence, we report αc\alpha_c, VcV_c, and McM_c against training time tt with fixed σ=1\sigma=1 using five seeds (Appendix C.1). Complementary grid size GG sweeps use both fixed strengths σ=1\sigma=1 and 1.51.5, also with five seeds (Figure 8).

In Fig. 8, at approximately 5,000 updates, both strengths reduce mean McM_c at every grid size. For σ=1\sigma=1, the reduction holds in all 50 paired comparisons. The results agree with the reviewer's expectations; VcV_c can increase, yet it is offset by αc2\alpha_c^2, demonstrating reduced population-level malign memorization for CURB-trained models.

Q5a) Extended Darcy comparison and MNIST comparison retain the trends in Table 2 with five seeds

On Darcy, CURB achieves 8.09±0.20%8.09\pm0.20\% relative L2L_2 error versus SAM’s 16.89±0.50%16.89\pm0.50\%, with lower error in every paird seed. We also included ASAM, which attained 12.65±0.24%12.65\pm0.24\% relative L2L_2 error. On MNIST, CURB achieves 89.41±0.30%89.41\pm0.30\% accuracy and improves over ERM in every evaluated configuration. The MNIST extension cover all six methods and seven configurations, totaling 210 runs, with the original experimental settings and training budgets retained.

Q5b) Hyperparameter selection

Appendix C documents the hyperparameter selection and experimental settings. The original smooth-regression CURB search was capped at σ=1\sigma=1; extending it to 1.5,2,3,5\\{1.5,2,3,5\\} selects σ=2\sigma=2 by independent clean validation. CURB then attains test MSE 0.0214±0.00700.0214\pm0.0070, making it competitive with SAM and ASAM. We report this in-text while retaining the original smooth-regression row in Table 1 for consistency with the original HPO ranges.

Kind regards,
Authors

Reviewer_pwzCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how KANs memorize mislabeled data in the training set and how such memorization affects the model's generalization capability; it further proposes a curvature-guided regularization method, CURB, to mitigate this harmful memorization.

Critical strengths

The authors point to a phenomenon that is easily overlooked: Fitting corrupted labels faster does not necessarily mean causing greater damage to the clean-data population. A finer grid might make it easier to memorize incorrect labels, but the resulting impact may remain localized, concentrated near the corrupted samples rather than spreading across the entire input space. This is a more specific perspective than simply asking whether a KAN is overfitting.

Critical weaknesses

The primary methodological innovation of the paper lies in "curvature-guided basis perturbation"; however, it lacks controlled comparisons against other basis-level perturbation directions.

While CURB’s overall performance is competitive, it is notably weaker than SAM and ASAM in the context of smooth regression. For Darcy, the standout result is based on only a single experimental trial, and the various MNIST configurations presented in the main table also largely reflect single-run outcomes. The claims regarding a robust advantage across tasks should be rephrased.

Questions for authors

The authors observe that CURB reduces corrupted-label fitting and improves clean-test performance. However, there is still a lack of sufficient evidence to confirm whether the former directly causes the latter?

1 comment
AuthorsOpenReview ↗

Author rebuttal

Dear Reviewer pwzC,

We are grateful for your praise on our specific perspective on the easily overlooked phenomenon.
We apologize for the late response.

Below we address the stated questions and concerns. In the revised PDF, we tagged the relevant parts with same enumeration and green, bold text in square brackets.

Q1) Controlled comparisons against other basis-level perturbation directions

We include direct controls supporting the curvature-guided design, with the clearest gains at high G/pG/p. The comparison replaces curvature perturbations with random perturbations, first derivatives or third derivatives, retaining the initialization, corrupted data, minibatch order and training budget. Random perturbation denotes uniform basis-value noise scaled to match CURB's expected integrated perturbation energy per atom. The MNIST comparison covers seven parameter-matched allocations and three paired seeds at 150 epochs, totaling 84 training runs.

PerturbationAverage MNIST accuracy (%)
CURB89.34±0.3089.34\pm0.30
Random perturbation87.00±1.6187.00\pm1.61
First derivative88.37±0.4788.37\pm0.47
Third derivative87.80±0.4387.80\pm0.43

Values are means and sample standard deviations over three seeds, after averaging the seven allocations within each seed; the third-derivative average includes the zero-direction quadratic case.

CURB's advantage over random perturbation is approximately 2 percentage points at (G,p)=(6,12)(G,p)=(6,12), approximately 4 points at (14,4)(14,4), and more than 4 points at (16,2)(16,2). At (14,4)(14,4), CURB achieves 89.42±0.51%89.42\pm0.51\%, versus 85.50±3.34%85.50\pm3.34\% for random perturbation and 85.99±1.69%85.99\pm1.69\% for first derivatives. CURB and first derivatives are nearly tied at the low-G/pG/p allocation (3,15)(3,15). Third derivatives are promising at higher degrees but vanish for p≤2p\le2, while curvature perturbations remain active for quadratic splines.

Q2) CURB achieves competitive performance on smooth regression with extended HPO

We thank the reviewer for pointing the smooth regression case. In fact, the reduced performance for smooth regression is due to the strength search being capped at σ=1\sigma=1. Extending it to 1.5,2,3,5\\{1.5,2,3,5\\}, while retaining the same 600 epochs and three paired seeds, selects σ=2\sigma=2 by independent clean validation. CURB now achieves 0.0214±0.00700.0214\pm0.0070 test MSE: 7.8% lower than SAM (0.0232±0.00160.0232\pm0.0016) and 4.6% higher than ASAM (0.0205±0.00120.0205\pm0.0012). Thus, CURB improves upon SAM’s mean performance and remains competitive with ASAM on this target.

We retain the original smooth-regression row in Table 1 for consistency with the original experimental protocol, and report the extended search result in the main text.

Q3) Extended Darcy and MNIST results confirm CURB’s advantage is robust.

We extended the Darcy comparison and the complete MNIST Table 2 to five seeds, retaining the original experimental settings and training budgets.

On Darcy, CURB achieves 8.09±0.20%8.09\pm0.20\% relative L2L_2 error versus SAM’s 16.89±0.50%16.89\pm0.50\%, with lower error in every paired seed. We also include ASAM, which achieves 12.65±0.24%12.65\pm0.24\% relative L2L_2 error.

On MNIST, CURB achieves 89.41±0.30%89.41\pm0.30\% accuracy versus ERM’s 72.91±0.45%72.91\pm0.45\%, improving mean accuracy in all seven configurations. It remains competitive with SAM (90.03±0.15%90.03\pm0.15\%) and ASAM (89.18±0.10%89.18\pm0.10\%). The extended Table 2 covers 210 total runs across six methods. All results are reported as means ± sample standard deviations.

Q4) New intervention experiments clarify the causal connection between reduced corruption fitting and improved generalization.

To test this causal link, we add extra fitting loss at corrupted training observations while keeping CURB active at its original strength (Figure 11). On the smooth 2D regression task with 20% corrupted labels, the intervention raises corruption fitting from 61.4% to 97.5%. Clean-test damage consequently increases from 0.0603 to 0.1160, removing CURB’s advantage in all three paired seeds.

We apply the identical intervention to ERM and include matched clean-label controls. The increase in damage is substantially larger under CURB than under ERM, which already fits the corrupted labels fully. The intervention strength is selected using training corruption fitting alone. These results provide strong evidence for the causal contribution of corruption-fitting suppression to CURB’s generalization benefit.

Kind regards,
Authors

Reviewer_8c7fOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the grid size G and spline degree p of B-spline KANs control the damage caused by fitting corrupted labels. In the NTK regime, the authors decompose the added population error from one corrupted sample as M_c = α_c² V_c. Here α_c is how much of the corruption is fitted at the corrupted point, and V_c is the population footprint of the fitted response.

In a localized two-layer regime they derive: Fitting rate: proportional to G p⁻² + p^{−1/2}, so finer grids fit corruptions faster. Footprint: V_c ∼ √p/G, so finer grids give narrower spikes. Net damage: M_c ∼ α_c² √p/G.

This explains why fixed-time comparisons make finer grids look more malign, while at matched corruption fit they are less so. The authors then propose CURB, which randomly perturbs each basis function along its normalized curvature direction during training. They show that it slows corruption fitting in the linearized model. Experiments cover synthetic regression, California Housing, Darcy flow, and MNIST/USPS with three basis families, with comparisons against SAM, ASAM, input noise and DropKAN.

Critical strengths

They address a clear question. Relating the KAN basis parameters to the malignancy of label-noise fitting is relevant, especially for scientific ML.

Separating fitting speed from footprint clarifies the issue. The distinction between fixed-time and matched-fit comparisons (Fig. 6) is a useful practical message that corrects the empirical literature.

The NTK analysis is clean and gives explicit (G, p) dependences through the B-spline kernel geometry (Lemmas 7–8). Working in the kernel regime is a reasonable choice that makes the analysis tractable.

The method CURB is simple, costs less than SAM, and works across bases and tasks, with large gains for Chebyshev and RBF KANs.

Critical weaknesses
  1. The regime of validity of the theory. Theorems 3–4 assume a localized regime in which the training Gram matrix is effectively diagonal, i.e. no other training point lies within kernel range of x_c. Because the first-layer kernel is additive over input coordinates, two points are coupled as soon as any single coordinate is within about (p+1)/G. In the experiments (N = 2048, G = 8–32), each point has many such neighbours. The response is then governed by K⁻¹ (the “equivalent kernel”) rather than by the bare kernel width. Could the authors state the condition on (N, d, G, p) for the localized regime, and discuss or measure how V_c behaves outside it? Fig. 5 should state explicitly in which regime it was computed.

  2. The frozen-NTK assumption at finite width under AdamW should also be checked, for instance by comparing measured α_c(t) and V_c with the kernel predictions in a trained network.

3)What CURB actually does. Theorem 6 shows that, after averaging over the noise, CURB adds a positive semidefinite quadratic penalty σ²uᵀCu, in line with the known equivalence between noise injection and Tikhonov regularization (Bishop, 1995). Since the ξ_k are independent, this penalty is diagonal in the coefficients and, roughly, a density-weighted weight decay. It is therefore unclear how much the curvature direction matters compared with plain weight decay.

  1. Table 2 and the Darcy result each come from a single repetition. On the smooth regression target, SAM and ASAM outperform CURB, by a wider margin at larger G (Table 6). More repetitions would make the claims of “competitive or superior” performance more convincing. Terminology.
Questions for authors

Why using the term "memorization"? “Memorization” in the work means malign overfitting of corrupted labels. It differs from memorization of training samples in, for example, generative models. A short clarification would help.

2 comments
AuthorsOpenReview ↗

Author rebuttal (1/2)

Dear Reviewer 8c7f,

We thank you for your detailed reading and finding our research question clear and solution simple, fast and general.
We apologize for the late response.

Below we address the stated questions and concerns. In the revised PDF, we tagged the relevant parts with same enumeration and red, bold text in square brackets.

Q1,Q2) Scope of the scaling laws and the footprint outside the localized regime

The effective regime of our theory follows from the frozen-NTK approximation, justified in the standard infinite-width limit. There is no universal numerical cutoff on NN or dd. We require the kernel KK to be invertible and approximately diagonal, which depends on sufficient sample separation relative to the spline resolution. This is the localized regime used in Appendix B through K−1ec≈ec/KccK^{-1}e_c\approx e_c/K_{cc}. For the joint degree–grid scaling, we additionally assume 1≪p≪G1\ll p\ll G, as stated in the statements of Theorems 3–4, yielding Eqs. (12)–(13).

We directly test the approach to this regime using two-layer [2,64,1][2,64,1] KANs with p=2p=2 on N=32N=32 inputs in two dimensions, leading to 32×3232\times 32-sized Gramians. Increasing GG isolates the effect of spline resolution relative to sample spacing. We compute the full-Gramian response to changing one training label and evaluate its population footprint on 8,192 independent inputs, averaging four interior corruption locations within each of three seeds.

| GG | Off-diagonal/mean-diagonal (∣K−diag(K)∣op/meaniKii\\|K-diag(K)\\|_{op}/mean_i K_{ii}) | First-layer share of diagonal | Full-Gramian VcV_c | Analytical VcV_c | |--:|--:|--:|--:|--:| | 32 | 0.9069 | 79.3% | 0.000726 | 3.2e-4 | | 64 | 0.2453 | 88.7% | 0.000209 | 1.6e-4 | | 128 | 0.0664 | 94.0% | 8.94e-05 | 8.1e-5 | | 256 | 0.0157 | 96.9% | 4.38e-05 | 4.0e-5 |

As grid resolution increases, off-diagonal interactions diminish from 0.9070.907 at G=32G=32 to 0.01570.0157 at G=256G=256 and the full-Gramian footprint approaches the analytical prediction, reaching approximately 9% discrepancy at G=256G=256.

Outside the localized regime, VcV_c depends on the full kernel response rather than the diagonal approximation. Appendix C.3 (Fig. 16) compares these full frozen-tangent predictions with nonlinear GD and AdamW training.

Q3) Kernel predictions closely follow GD and AdamW dynamics

Following the reviewer’s suggestion, we include kernel comparisons (Appendix C.3, Fig. 16) for GD and AdamW using the smooth 2D regression task over three seeds at G=16,32G=16,32. The predictions closely follow the seed-averaged dynamics. Under GD, endpoint discrepancies remain below 1.3% for αc\alpha_c and 0.21% for VcV_c at both grids. Under AdamW at G=32G=32, αc\alpha_c differs by only 0.0012% and VcV_c by 1.7%, while for G=16G=16, αc\alpha_c differs by 5.9% and VcV_c by 43.5%.

For GD, we use the standard discrete-time NTK prediction. For AdamW, we optimize the frozen-tangent linearization with the evolving adaptive moments to address the adaptive tangent dynamics [1]. Neither comparison refits the kernel or introduces fitted constants.

We think the departure in AdamW, G=16G=16 is caused by feature evolution during the longer fitting transient where the parameter Jacobian changed by approximately 20%, compared with 8% at G=32G=32 which is consistent with the dependence of frozen-tangent accuracy on Jacobian stability [2].

Q4a) The fixed quadratic does not reproduce CURB’s benefit

We directly optimize

MSE(θ)+σ2(θ−θ0)TC(θ0)(θ−θ0),\mathrm{MSE}(\theta)+\sigma^2(\theta-\theta_0)^T C(\theta_0)(\theta-\theta_0),

using the exact all-layer operator frozen at initialization. On oscillatory 2D regression, CURB achieves 0.01792±0.002790.01792\pm0.00279 clean-test MSE, versus 0.11331±0.007670.11331\pm0.00767 for the matched-strength quadratic and 0.09601±0.006950.09601\pm0.00695 after validation selection (three paired seeds; Appendix C.4, Table 6).

Thus, the fixed local quadratic does not capture CURB’s benefit throughout nonlinear training. We also tested averaging the full perturbed loss by averaging eight perturbation losses per update to simulate CURB's effect, which retained the benefit, achieving 0.02054±0.003740.02054\pm0.00374.

We demonstrate in Appendix C.4 that the atom noise does not make the induced parameter operator diagonal: shared learned coordinates produce off-diagonal terms.

We additionally tested weight decay over a broad range extending to wd=100wd=100; on California Housing, validation selected wd=10wd=10, reducing MSE relative to ERM, yet CURB's MSE remained 16.0%16.0\% lower than tuned weight decay.

AuthorsOpenReview ↗

Author rebuttal (2/2)

Q4b) The curvature direction matters beyond the presence of random basis noise

To investigate how much the curvature direction matters, we include matched-energy comparisons against random perturbation, first derivatives and third derivatives (Appendix C.2). Random perturbation denotes uniform basis-value noise scaled to match CURB’s expected integrated perturbation energy per atom. The MNIST comparison covers seven parameter-matched (G,p)(G,p) allocations as in Table 2, using three paired seeds. Each run uses 150 epochs, totaling 84 training runs.

PerturbationAverage MNIST accuracy (%)
CURB89.34 ± 0.30
Random perturbation87.00 ± 1.61
First derivative88.37 ± 0.47
Third derivative87.80 ± 0.43

Values are means and sample standard deviations over three seeds, after averaging the seven allocations within each seed; the third-derivative average includes the zero-direction quadratic case.

CURB-training attains better average accuracies. The clearest advantages for CURB appear in the high-G/pG/p regime. Its gain over random perturbation increases from approximately 2 percentage points at (6,12)(6,12) to approximately 4 points at (14,4)(14,4) and more than 4 points at (16,2)(16,2). At (14,4)(14,4), CURB achieves 89.42±0.51%89.42\pm0.51\%, versus 85.50±3.34%85.50\pm3.34\% for random perturbation and 85.99±1.69%85.99\pm1.69\% for first derivatives. At low G/pG/p, CURB and first derivatives are nearly tied at (3,15)(3,15). These controls complement the explicit-quadratic comparison in Q4a.

Third derivatives are promising at higher degrees but vanish for p≤2p\le2, producing no perturbations, in which case no regularization is applied.

Q5a) CURB remains superior on Darcy and competitive on MNIST with 5 seeds.

On Darcy, CURB achieves 8.09±0.20%8.09\pm0.20\% relative L2L_2 error versus SAM’s 16.89±0.50%16.89\pm0.50\%, outperforming SAM in every paired seed. On MNIST, CURB achieves 89.41±0.30%89.41\pm0.30\% accuracy, remaining competitive with SAM’s 90.03±0.15%90.03\pm0.15\%.

Q5b) Extending CURB’s strength search makes smooth regression performance competitive with SAM and ASAM

We include an extended HPO study on the smooth 2D regression task, the original CURB search was capped at σ=1\sigma=1; we additionally evaluated σ∈1.5,2,3,5\sigma\in\\{1.5,2,3,5\\}, retaining the architecture, data, optimizer schedule and training budget. CURB then achieves clean-test MSE 0.0214±0.00700.0214\pm0.0070, compared with 0.0232±0.00160.0232\pm0.0016 for SAM and 0.0205±0.00120.0205\pm0.0012 for ASAM. This reduces CURB’s original MSE of 0.0488±0.00240.0488\pm0.0024 by 56.2%, giving lower mean error than SAM and performance close to ASAM. Appendix C documents the original experimental protocols.

Q6) Memorization terminology follows the related literature.

Following earlier works [3,4], we use “memorization” to describe fitting corrupted training labels, and "malign memorization" when this harms generalization. We agree with the reviewer that "malign overfitting to corrupted labels" accurately describes the phenomenon studied here. We retain "malign memorization" for consistency with the related literature.

Kind regards,
Authors

[1]: Littwin and Yang. Adaptive optimization in the $\infty $-width limit. ICLR 2023
[2]: Chizat et al. On lazy training in differentiable programming. NIPS 2019
[3]: Anagnostidis et al. The curious case of benign memorization. ICLR 2023
[4]: Bayat et al. The pitfalls of memorization: when memorization hurts generalization. ICLR 2025

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.