acceptodds
Under review as a conference paper at ICLR 2027

Action at a Distance: A Universal Kernel from Polynomial Alignment and IMQ Distance

Abstract

Inverse-multiquadric (IMQ) kernel sections decay at infinity. We study how quadratic alignment balances their inverse-square decay, adding persistent directional responses while preserving compact-domain approximation. The Yat kernel multiplies the IMQ kernel by a squared biased inner product. For fixed positive bias and distance offset, its RKHS continuously contains the IMQ RKHS and is universal on every compact domain. This global RKHS inclusion is strict: Yat sections with nonzero centers have a nonzero quadratic directional trace at infinity. Alignment also changes high-dimensional activation covariance. For a fixed number of orthogonal unit centers and fixed kernel parameters, isotropic Gaussian inputs give a rank-one rescaled limiting covariance for IMQ and a full-rank limit for Yat. Under uniform inputs on the radius- sphere, both limits are full rank, identifying shared input-norm fluctuations as the Gaussian collapse mechanism in this regime. The connection is constructive: three positive-bias Yat atoms recover any IMQ section exactly, and three are necessary at nonzero centers when the distance offset is shared and biases may vary across atoms. On bounded cubes, rank- positive-semidefinite quadratics admit -atom Yat approximations with growing center scales. Accurate IMQ approximation obeys explicit necessary coefficient-variation budgets for , without restrictions on width or center locations. Polynomial alignment thus preserves compact-domain approximation while changing global directional behavior and the resources used to represent functions.

Primary area:
learning theory
Keywords:
kernel methods, reproducing kernel Hilbert spaces (RKHS), universal approximation, activation functions, approximation theory, representation learning, high-dimensional geometry, inverse multiquadric kernels

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zynPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the Yat Kernal, which is the product of the degree-two polynomial kernel and the inverse-multiquadric (IMQ) kernel. For fixed positive bias and fixed distance offset, the Yat function space contains the IMQ function space and can approximate continuous functions on compact input domains. Yat sections also preserve direction-dependent behavior at infinity. The directional trace of finite expansions cannot be extended continuously to the completed Yat function space.

Theorem 1 compares feature covariance under Gaussian and fixed-norm inputs. Theorems 2 and 3 establish exact IMQ recovery with sharp three-atom complexity, its minimality at nonzero centers sharing a distance offset, a low-width quadratic approximation, and necessary budgets on IMQ coefficients.

Critical strengths
  • This kernal theory paper features a detailed controlled covariance comparison, as the authors summarized in the second paragraph of section 6. The calculation and its moment justification look sound, making this the strongest contribution of this paper.

  • The paper also provided specific representation analysis.

Critical weaknesses
  • The theoretical advance is not deep and not well positioned against prior work. The basic zero-bias Yat construction predates the submission, and several guarantees are standard kernel arguments; the covariance and representation results are useful but narrow. Three-atom minimality concerns exact equality with a shared distance offset, whereas one growing-bias atom can uniformly approximate an IMQ section. The paper should clarify its specific new increment and cite the earlier construction.

  • The approximation comparison leaves the cost tradeoff unresolved: Yat bounds unit count while center lengths and kernel norms grow, whereas IMQ bounds total absolute coefficients, so comparable function-norm efficiency is not established. The authors correctly claim neither a matching width lower bound nor improved generalization; experiments add little weight given unmatched tuning, mixed zero-shot gains, and unverifiable records.

  • The paper is hard to read, with many nonations weren't properly defined before introduction.

Questions for authors
  1. Is there any further practical motivation of developing this Yat kernal theory, beyond answering the pure mathematical question of "when we add a directional component to a distance-based kernel, what properites change and what are preserved", which is a reasonable mathematical question by itself?

  2. How does the construction relate to arXiv:2411.08085? Please cite and compare the earlier zero-bias operation and neuron use, and identify what is genuinely new in the current work.

  3. Can the language-model experimental results be verified or reproduced?

  4. Is there a concrete finite-error setting in which the paper’s coefficient thresholds and Yat construction yield a useful tradeoff once center lengths and kernel norms are taken into account?

1 comment

Thank you for the careful reading. We address your concerns below.

1. Practical motivation. Along a ray with nonzero alignment, ⵟ’s response retains sensitivity to its tied center. Proposition 7 gives a nonzero tied-center gradient limit for ⵟ, while IMQ's center gradient decays as Θ(r−3)\Theta(r^{-3}). The ⵟ input gradient still decays as O(r−1)O(r^{-1}). Unit derivatives do not guarantee loss gradients. Amirian and Schwenker discuss attenuated RBF gradients for inputs initially far from centers. Amirian and Schwenker, §IV

ⵟ FFNs trained in all 12 blocks of a 261M Transformer on 5.22B C4 tokens. Shared bias with learned scale gave 39.07 ± 0.39 WikiText perplexity versus GELU’s 44.23 ± 1.44 (three seeds, mean ± SD), at 655K versus 715K tokens/s (Table 4). Zero-shot results were mixed (Table 5). Our exploratory IMQ run at that scale collapsed (Appendix F.4). Equal tuning budgets and the center gradient’s role remain open.

On the displaced disk, moving the distance center with alignment fixed lowered relative MSE from 80.96% to 66.16%. The best fit, 29.65%, used a separate 2D center. The radial Gaussian reached 51.15%, ahead of both tied fits (52.26% centered, 56.42% Eq. 1) (§3.4, Appendix F.5, Table 6).

2. What changed from earlier work? This study grew out of the 2024 preprint’s zero-numerator-bias ⵟ neuron, with a learned scale and a separate output bias. The 2025 preprint established zero-bias positive semidefiniteness and the single-atom directional limit. We discuss both in §1 and Appendix A.3. Bouhsine (2024), §3.1 · Bouhsine (2025), v2, Apps. G.2–G.3

At b=0b=0, we have universality on compacts excluding zero. On compacts containing zero, the uniform RKHS closure consists of continuous functions vanishing there. An output bias restores density by approximating f−f(0)f-f(0) and adding f(0)f(0), leaving the kernel unchanged (Proposition 4, Appendix B.2). Here bb shifts the inner product before squaring. For b>0b>0, b2hε⪯kb,εb^2h_{\varepsilon}\preceq k_{b,\varepsilon} gives IMQ RKHS inclusion and universality on all compacts (§2). Aronszajn, §7 · Micchelli, Xu, and Zhang, Theorem 17

Our additions include strict global RKHS inclusion for b>0b>0, Gaussian and spherical covariance limits, and a finite-span trace with no continuous RKHS extension (§3). At a nonzero center, three variable-bias atoms with a shared offset are necessary for exact IMQ recovery (Theorem 2). For uniform approximation, one scaled atom suffices as its bias grows (Lemma 5).

3. Reproducibility. Code, configs, and tokenizer are in the supplement. GELU seeds 0 1 2. Shared-bias ⵟ seeds 0 1 2. Histories reconstruct all 40 reported cells. From each seed’s first complete pass, WikiText perplexity is 39.06 for learned-scale shared-bias ⵟ and 44.24 for GELU. Historical training and evaluator revisions, scored checkpoints, and long-context data remain unpinned (Appendix F.4).

4. What do the shared budgets buy? In Theorem 3 (§4.2), the shared budget is the sum of absolute coefficients on raw atoms at fixed offset. See Bach, §2.1. For a PSD quadratic with matrix AA, ⵟ reaches any accuracy with at most rank⁡(A)\operatorname{rank}(A) atoms at budget tr⁡(A)\operatorname{tr}(A). Every fixed IMQ budget leaves positive uniform error even with unrestricted width and centers. Otherwise Montel gives a bounded holomorphic limit equal to an unbounded quadratic along a line. We leave the divergence rate open. The ⵟ construction uses growing centers and an RKHS norm of order δ−2\delta^{-2} at error δ\delta. Matched-width or matched-norm advantages remain unproved.

For a constructed example on 10≤∥x∥≤2010\le\|x\|\le20, take g=k1,1(w,⋅)/2g=k_{1,1}(w,\cdot)/2 for unit ww. Cap center radii, raw coefficient variation, and the common ambient H1,1\mathcal H_{1,1} norm at one. One ⵟ atom fits exactly. The IMQ coefficient cap implies the norm cap by kernel order. At 10w10w, g(10w)=121/164g(10w)=121/164 and every admissible IMQ fit has magnitude at most 1/821/82, giving uniform error at least 119/164119/164. This is an existence example under those caps.

5. Readability. We added a same-projection, different-distance example near Eq. (1). §2 and Table 1 collect notation; Appendix A.1 gives definitions. Appendices C.2.1 and D.3 explain when bounds cease to be informative and why. The conclusion explains the ablation and states that its best fit uses a separate center.

Reviewer_PCQyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the Yat kernel, which combines polynomial alignment with an inverse-multiquadric (IMQ) distance factor: $
k_{b,\varepsilon}(w,x) =\frac{(w^\top x+b)^2}{|x-w|^2+\varepsilon}, \qquad b\geq 0,\ \varepsilon>0. $

The paper investigates the kernel's theoretical properties, including positive definiteness, RKHS approximation properties, asymptotic behavior at infinity, and high-dimensional activation covariance. It also establishes an exact finite-atom identity connecting a family of Yat kernels to the IMQ kernel and studies approximation properties for positive-semidefinite quadratic forms.

The most interesting theoretical result concerns high-dimensional activation covariance. Under the paper's stated assumptions, IMQ activations exhibit a rank-one leading-order covariance structure for isotropic Gaussian inputs, whereas the proposed polynomially aligned kernel can retain a richer limiting covariance structure. The analysis of fixed-norm spherical inputs further clarifies that the covariance phenomenon depends on the input distribution and assumptions.

The paper also includes empirical studies of activation statistics, far-field approximation, and language-model performance. These experiments provide preliminary evidence about the proposed kernel's behavior, although they do not yet establish that its theoretical structural advantages consistently translate into better optimization, generalization, or extrapolation.

Critical strengths
  1. An interesting theoretical analysis of high-dimensional activation covariance: The covariance analysis is the paper's strongest contribution. Rather than relying solely on standard kernel properties, it characterizes the asymptotic covariance structure of kernel activations under specified high-dimensional input distributions. The contrast between Gaussian and fixed-norm spherical inputs is particularly useful because it helps delineate the conditions under which the reported degeneracy occurs. This analysis provides a potentially valuable perspective on the design and interpretation of kernel-based neural units.

  2. A coherent characterization of the proposed kernel: The paper examines several complementary properties of the Yat kernel, including positive definiteness, RKHS relationships, directional behavior at infinity, and finite-atom representations. Although some individual arguments follow from established kernel-theoretic tools, their combination provides a reasonably coherent account of how polynomial alignment changes the behavior of an IMQ-based kernel.

  3. A clear distinction can be made between standard consequences and kernel-specific results: The positive-definiteness and universality arguments appear to rely substantially on established results, such as closure properties of positive-definite kernels and RKHS inclusion arguments. By contrast, the high-dimensional covariance asymptotics and the directional far-field characterization are more specific to the proposed construction. This gives the paper a plausible theoretical contribution beyond merely introducing another kernel formula.

  4. The paper examines both theoretical behavior and practical implementations: The inclusion of activation-statistics experiments and language-model experiments is a strength. In particular, the reported language-model results suggest that the proposed activation may be practically competitive under the tested setup. While the current experiments do not establish broad superiority, they provide a useful starting point for evaluating the practical relevance of the theoretical analysis.

Critical weaknesses
  1. The connection between the theoretical results and practical learning benefits remains underdeveloped: The paper's theoretical analysis establishes structural properties of the proposed kernel, but these properties do not by themselves imply better optimization, generalization, or extrapolation. For example, a richer limiting activation covariance does not necessarily guarantee improved task performance, and the far-field behavior of individual kernel atoms does not directly establish superior out-of-distribution prediction. The paper would be more convincing if it more carefully separated the established mathematical results from the practical implications that remain hypotheses.

  2. The novelty of some theoretical results is limited: Several results, including positive definiteness and aspects of the RKHS approximation analysis, follow from standard kernel-theoretic principles. The exact finite-atom identity is useful, but its algebraic form is relatively simple. The paper's originality therefore rests more heavily on the high-dimensional covariance analysis and the combined characterization of the proposed kernel than on each theorem considered individually. A clearer explanation of the precise boundary between established theory and genuinely new results would strengthen the contribution.

  3. The empirical evidence is not yet sufficient to validate the central practical claims: The activation-statistics experiments support aspects of the proposed structural analysis, but they do not directly demonstrate improved predictive performance. In the reported CLIP activation diagnostics, the effective-rank results do not consistently reproduce the idealized full-rank behavior suggested by the orthogonal-center analysis. Likewise, the far-field approximation experiments do not establish that the proposed kernel leads to better extrapolation after comparable high-accuracy fitting. The language-model results are promising, but the comparison leaves questions about tuning fairness and efficiency. In particular, the reported learning rates differ between the Yat and GELU configurations, and the Yat models have lower throughput. The available evidence therefore does not yet isolate the effect of the activation function from other training choices or establish a consistent advantage across tasks.

  4. Some approximation-complexity comparisons use different resource measures: The paper compares the number of Yat atoms with a coefficient-total-variation budget for IMQ expansions. These are not directly equivalent measures of approximation cost. In addition, the Yat construction may require changes in center scale or norm as the target approximation error decreases. Consequently, the current results support particular representation and lower-bound statements, but they do not yet justify a broad conclusion that Yat is uniformly more resource-efficient than IMQ.

  5. Reproducibility and evaluation details need clarification: The reported language-model experiments raise questions about hyperparameter selection, evaluation-data provenance, and the completeness of evaluation records. These issues do not invalidate the theoretical results, but they reduce confidence in the empirical comparison and make it harder to determine whether the reported gains are robust.

Questions for authors
  1. Scope of the high-dimensional covariance result: Could the authors state the precise assumptions required for the limiting covariance results, including the scaling of the bias parameter, the norms and mutual geometry of the centers, and the input distribution? In particular, which aspects of the rank-one IMQ covariance behavior persist beyond the specific isotropic Gaussian setting, and which are specific to the stated asymptotic regime?

  2. Fairness of the language-model comparison: The reported Yat and GELU experiments use different learning rates, and the Yat configurations exhibit lower throughput. Were both activation functions tuned under comparable hyperparameter-search budgets and training protocols? Could the authors provide evidence that the perplexity improvements persist under a matched tuning procedure and clarify the trade-off between predictive performance and computational efficiency?

  3. Reproducibility of the long-context evaluation: Could the authors clarify the provenance of the long-context evaluation data and provide sufficient details about the evaluation pipeline, data splits, and evaluation implementation to reproduce the reported results? Would the reported performance differences remain under an independently reproducible evaluation setup?

  4. Interpretation of the approximation-complexity comparison: Could the authors clarify the intended interpretation of the comparison between the number of Yat atoms and the coefficient-total-variation budget for IMQ expansions? Is there a comparison under a common resource measure, such as approximation error versus the number of atoms or a comparable norm budget, that supports the claimed efficiency advantage?

  5. Connecting structure to learning behavior: Can the authors identify which practical claim they consider most directly supported by the current theory: improved optimization, generalization, or extrapolation? Further experiments with controlled training protocols could help establish whether the covariance and far-field properties predict measurable downstream improvements. This would strengthen the practical interpretation of the results, although the absence of such experiments alone would not invalidate the mathematical contributions.

2 comments

Thank you for the close reading and for asking what the theory tells us about learning.

1. Covariance: assumptions and scope. Theorem 1 fixes the number of unit centers mm, bias b≥0b\ge0, and offset ε>0\varepsilon>0 as d→∞d\to\infty, with center Gram limit CC. For isotropic Gaussians, IMQ covariance scaled by d3d^3 tends to 211⊤2\mathbf{1}\mathbf{1}^\top. With m≥2m\ge2 orthonormal centers, common and contrast eigenvalues scale as 2m/d32m/d^3 and 4/d44/d^4.

At fixed input norm d\sqrt d, IMQ covariance scaled by d4d^4 tends to 4C4C. For ⵟ, the d2d^2-scaled limit is 2(C∘C)+4b2C2(C\circ C)+4b^2C under both laws, also given by the squared numerator alone. On a sphere the classes coincide with matching centers and offset, a linear skip, and an intercept (§3.3). Frozen CLIP gives similar effective ranks (Appendix F.2). Extensions to other input laws remain open.

2. LM comparison and cost. Both 261M, 12-block models used 5.22B tokens and the same AdamW schedule (500-step warmup, cosine decay, batch 256). The best peak rates from separate short sweeps were 0.01 for GELU and 0.03 for ⵟ. Shared-bias ⵟ with learned scale gave 39.07 ± 0.39 WikiText perplexity versus 44.23 ± 1.44 for GELU (three-seed mean ± SD), at 655K versus 715K tokens/s on TPU v6e-8. Zero-shot results were mixed (Tables 4–5). Equal tuning budgets and equal-time comparisons remain open. The exploratory IMQ run collapsed at that scale (Appendix F.4). Dodge et al., §3 · Mattson et al., §3.2

3. Long-context evaluation. We can reconstruct the scores, but cannot yet identify the exact long-context dataset and scored revisions. Logs show 16,380 scored tokens. The recovered evaluator uses four nonoverlapping 4,096-token chunks and tries PG-19 validation, PG-19 test, then WikiText-103 test. We reconstructed 40 cells from 15 histories using the second complete pass for seed 0 and the first for seeds 1–2. The historical selection rationale is unknown. First passes throughout preserve the learned-scale WikiText ordering and reverse it for constant scale. Exact dataset, split, checkpoint, and evaluator revisions remain unpinned. A rerun would pin these along with the tokenizer and windows (Appendix F.4). Rae et al., §4.2 · Press, Smith, and Lewis, §3

4. Approximation budgets. Theorem 3 budgets raw-atom coefficient variation at fixed offset. For a rank-rr PSD quadratic, ⵟ uses at most rr atoms and variation tr⁡A\operatorname{tr} A at every accuracy. IMQ has a positive error floor at that budget even with unrestricted width and centers. Montel gives budget divergence as δ↓0\delta\downarrow0. The rate is open. Our centers and RKHS norm grow, the latter as δ−2\delta^{-2}. Unit RKHS normalization raises coefficient variation. Matched-width or matched-norm advantages remain unproved. Bach, §2.1

On the displaced disk target with a Gaussian envelope (§3.4, Appendix F.5), the near-zero-bias Eq. (1) atom reaches 56.42% relative population MSE versus 51.15% for a radial Gaussian. In the frozen-numerator ablation, moving the tied center lowers error from 80.96% to 66.16%. The best fit, 29.65%, adds a separate center.

5. What follows for learning? For centered frozen features, covariance is the population half-squared-loss Hessian. With at least two orthonormal centers, its Gaussian condition number diverges for IMQ and tends to one for ⵟ. Both tend to one on the sphere. This controls worst-case contraction for readout gradient descent at each model’s optimal fixed step. End-to-end effects remain open. Polyak, Theorem 9

Along aligned rays, ⵟ's tied-center gradient has a nonzero limit (Proposition 7), IMQ's decays as Θ(r−3)\Theta(r^{-3}), and ⵟ's input gradient decays as O(r−1)O(r^{-1}). These are unit derivatives, not a guarantee about the full loss gradient. The 200-center fixed-ring IMQ fit reached annulus MSE 3.447×10−63.447\times10^{-6} and tail contrast 0.00257 versus the target's 1.008948, with a larger offset and more coefficient variation than the jointly trained fits (Appendix F.3). This remains a synthetic extrapolation diagnostic; matched-error comparisons with jointly trained fits remain open.

PSD and RKHS inclusion use classical results and the recovery identity uses finite differences. The sharp three-atom minimum for exact IMQ recovery at nonzero centers with shared offset, covariance limits, and finite-span trace are specific to this kernel. Aronszajn, §§7–8

The authors have made substantial and commendable revisions in response to the reviewers' concerns. In particular, the revised covariance analysis rigorously delineates its underlying assumptions, clearly distinguishing between Gaussian and fixed-norm inputs. The newly added fixed-ring IMQ experiment provides compelling evidence for the directional-tail distinction, and the updated approximation analysis establishes a much more precise comparison under a shared raw coefficient-variation budget. Furthermore, the authors have significantly enhanced the transparency of the language-model experiments and proactively documented the limitations of the historical evaluation records. 

These revisions strengthen the mathematical presentation and clarify the scope of the theoretical claims. Nevertheless, several critical concerns remain unresolved. First, the practical connection between the theoretical covariance/representation results and downstream learning benefits remains tenuous. Second, the language-model comparison still lacks fully matched hyperparameter-tuning budgets, and the long-context evaluation remains incompletely traceable regarding the exact dataset, data split, and checkpoint revisions. Third, the approximation advantage is demonstrated under a specific coefficient-variation measure, which does not fully extend to a broader matched-resource comparison.

Reviewer_eoWoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces the Yat kernel, which augments the inverse-multiquadric (IMQ) kernel with a quadratic biased inner product. The authors provide a detailed theoretical analysis showing that, for positive bias, the Yat RKHS contains the IMQ RKHS while exhibiting distinct global directional behavior and high-dimensional covariance structure. They also derive exact representation and approximation results and provide diagnostic experiments, including large-scale language-model feasibility tests.

Critical strengths
  • The paper studies the proposed kernel from several complementary perspectives, including positive definiteness, RKHS inclusion and universality, asymptotic directional behavior, high-dimensional covariance, finite-atom representation, and layer-level RKHS geometry. Many of the results go beyond simply establishing that the proposed product is a valid kernel.
  • The constructed kernel combines a standard IMQ distance factor with an explicit polynomial alignment factor, making it possible to retain familiar RKHS guarantees while introducing a qualitatively different directional response. The simplicity of the construction is appealing, and its properties can be analyzed in considerable detail.
Critical weaknesses

The main concern is that the practical or broader machine-learning value of the proposed Yat kernel remains unclear. The paper establishes many mathematically interesting differences between Yat and IMQ kernels, but it does not clearly demonstrate why these differences are important for solving realistic learning problems. For example, the nonvanishing far-field directional trace and the full-rank asymptotic covariance under Gaussian inputs are elegant properties, but the paper does not establish that either property leads to systematically better prediction, optimization, sample efficiency, robustness, or generalization in settings where existing kernels or standard neural activations are used. As stated in the paper, the experiments on large-scale language models are not matched-tuning comparisons and do not establish either a generalization advantage or a global depth-capacity guarantee. In my opinion, a convincing downstream setting in which the proposed geometry yields a clear and reproducible advantage, or a stronger theoretical result directly connecting the new geometry to a meaningful learning guarantee, would be needed to justify acceptance at a top-tier ML conference.

Questions for authors

N/A

2 comments

Thank you for reading the paper closely. Since you raised concerns rather than specific questions, we would like to clarify what the experiments show and what the geometry establishes about learning.

“Solving realistic learning problems.” Learned-center ⵟ FFNs trained in all 12 blocks of a 261M causal Transformer on 5.22B C4 tokens. With comparable parameter counts and token budgets, shared-bias ⵟ with learned scaling reached 39.07 ± 0.39 WikiText perplexity versus 44.23 ± 1.44 for GELU (mean ± SD over three seeds). Throughput was about 655K versus 715K tokens/s (Appendix F.4, Table 4). This establishes trainability at the tested depth and an observed perplexity improvement under these recipes. Zero-shot results were mixed (Table 5). Systematic learning gains and a global depth-capacity bound remain open.

We ran a short learning-rate sweep for each activation because changing the activation changes the gradients the optimizer sees. We used the best peak rate from each sweep, 0.01 for GELU and 0.03 for ⵟ. The question we haven't answered is whether the perplexity gap holds when both models get the same tuning budget. Ramachandran et al., §§5.3–5.4 · Hendrycks and Gimpel, §3.4 · Fang et al., App. C.1

“Full-rank asymptotic covariance under Gaussian inputs.” This has a precise learning consequence for a frozen readout. When the feature bank is frozen and centered, its covariance is the population half-squared-loss Hessian for a linear readout. With fixed orthonormal centers and Gaussian inputs, IMQ’s common mode scales as 2m/d32m/d^3 and the contrast eigenvalues as 4/d44/d^4 (Theorem 1). For m≥2m\ge2, IMQ’s condition number diverges while ⵟ’s approaches one, so gradient descent with each model’s optimal fixed step gives better worst-case contraction for ⵟ. The squared numerator already gives ⵟ’s leading rescaled covariance. On the sphere, both condition numbers approach one, and both families span the same functions given shared centers and offset, a linear skip, and an intercept (§3.3). Whether this conditioning advantage carries over when the features are learned end to end remains open. Le Cun, Kanter, and Solla, §3 · Polyak, Theorem 9

“Nonvanishing far-field directional trace.” This motivates asking whether a center can learn from distant inputs. Amirian and Schwenker describe weak gradients when inputs start far from their RBF centers. Proposition 7 makes a specific comparison. IMQ’s center-gradient norm decays as Θ(r−3)\Theta(r^{-3}). Along a ray with nonzero alignment, ⵟ’s tied-center gradient approaches the nonzero derivative of the trace with respect to the center. The ⵟ input gradient still decays as O(r−1)O(r^{-1}). The unit therefore retains center sensitivity, although a nonzero unit derivative alone does not ensure a useful end-to-end loss gradient. We have yet to determine whether it helped train centers in our LM. Amirian and Schwenker, §IV

Our exploratory inverse quadratic IMQ run at 261M collapsed (Appendix F.4). This single exploratory run leaves the comparison with systematically tuned IMQ at that scale open. Related RBF results show sensitivity to the recipe. DeepLABNet reports 3.55% CIFAR-100 accuracy with conventional RBF activations but 65.3% with its redesigned Gaussian variant. Soares et al. recover convergence in small complex RBF stacks using tailored initialization. Schliebitz et al. report collapse with IMQ vision projection heads, using exponent −1/2-1/2 rather than our −1-1. These studies do not explain our collapse or rule out tuned IMQ at our scale. DeepLABNet, Tables 2 and 5 · Soares et al., §IV · Schliebitz et al., §IV-B · Kernel implementation

For extrapolation, fixed-ring IMQ fits beat the annulus constant baseline but retain little directional contrast in the tail (Appendix F.3). This is a synthetic illustration of the trace obstruction, not evidence of better generalization on realistic tasks.

We would welcome your view on which downstream task or learning guarantee would most help clarify the value of this geometry.

Thank you for the detailed response and additional clarifications. I appreciate the authors’ efforts to address my concerns.

The theoretical result showing that the Yat kernel avoids the diverging condition number of IMQ, thereby improving the worst-case convergence rate of gradient descent, is certainly valuable. However, this result is established for a linear readout with fixed orthonormal centers and Gaussian inputs, which differs substantially from the settings of practical deep neural networks. While mathematically interesting, it does not yet establish a clear advantage in realistic learning problems. Also, the language-model results are promising, but the absence of a comparison with a sufficiently tuned IMQ baseline makes it difficult to determine whether the observed improvements arise from the distinctive geometric properties of the Yat kernel. Therefore, my primary concern regarding the practical machine-learning significance of the proposed kernel remains unresolved, and I will maintain my original recommendation.

Comments on the paper

1 comment

We made the following changes in the revised PDF.

  • We credit the 2024 arXiv preprint’s zero-bias neuron and the 2025 preprint’s proof of positive semidefiniteness and single-atom directional limit for the zero-bias kernel (§1 and Appendix A.3). We distinguish the earlier output bias from the numerator bias and clarify zero-bias approximation (Appendix B.2).
  • We clarify that our radial factor is also called the inverse quadratic kernel and give a same-projection example showing how distance to the center changes the response (§1, Eq. (1)).
  • We give the exact spherical identity and explain why fixed-center ⵟ and IMQ models with a free linear input skip and intercept represent the same functions even when their raw activation covariances differ (§3.3).
  • We derive the Gaussian IMQ contrast eigenvalues' 4/d44/d^4 scaling for orthonormal centers and show it in Figure 1 with the squared numerator as a control (§3.2 and Appendix F.1).
  • We added a controlled ablation that holds the alignment numerator fixed while varying the distance center (§3.4 and Appendix F.5). The conclusion explains that moving the center improves the fit and that the lowest error uses a separate center (§6). Table 6 also reports the original tied Eq. (1) atom with nonnegative bias, which reaches 56.42% relative MSE.
  • Near Proposition 7, we clarify that along a distant ray with nonzero alignment, a single ⵟ unit can retain a gradient with respect to its tied center while its input gradient still decays as O(r−1)O(r^{-1}).
  • We evaluate the covariance bounds at the dimensions in Figure 1 and report which Gaussian and spherical spectral conclusions they certify (Appendix C.2.1).
  • We add a Montel proof that fixed-offset IMQ requires diverging coefficient variation for increasingly accurate quadratic approximation. The shared budget counts coefficients of raw atoms. The ⵟ construction keeps that budget fixed as its centers and RKHS norm grow (§4.2, Theorem 3).
  • We add a fixed-ring IMQ least-squares check with 50 and 200 centers. Both fits beat the best constant on the annulus and recover little of the target's directional contrast on the tail (Appendix F.3).
  • Appendix F.4 clarifies that the 12-layer language-model runs compared ⵟ FFNs with GELU and used the best peak learning rate from each short sweep, 0.01 for GELU and 0.03 for ⵟ. The per-neuron-bias runs used 19,922 steps, and the other runs used 19,920. An exploratory IMQ FFN run at that scale collapsed during training. Appendix F.4 cites Schliebitz et al. for collapse in self-supervised vision projection heads and distinguishes their IMQ exponent of −1/2-1/2 from our inverse quadratic exponent of −1-1.
  • Appendix F.4 cites both DeepLABNet comparisons on CIFAR-100 (Tables 2 and 5). The conventional RBF ResNet-20 reached 3.55% validation accuracy, and the Gaussian variant in their redesigned architecture reached 65.3%. The appendix also cites Soares et al.'s convergence failures under two initializations in smaller complex-valued RBF stacks for a 5G simulation.

The 15 checkpoints below cover all five 261M configurations in Appendix F.4. We verified each Anonymous-HF link while signed out. The supplement includes the code, configs, and tokenizer for evaluating GELU and the learned-scale, shared-bias ⵟ variant.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.