- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the Yat Kernal, which is the product of the degree-two polynomial kernel and the inverse-multiquadric (IMQ) kernel. For fixed positive bias and fixed distance offset, the Yat function space contains the IMQ function space and can approximate continuous functions on compact input domains. Yat sections also preserve direction-dependent behavior at infinity. The directional trace of finite expansions cannot be extended continuously to the completed Yat function space.
Theorem 1 compares feature covariance under Gaussian and fixed-norm inputs. Theorems 2 and 3 establish exact IMQ recovery with sharp three-atom complexity, its minimality at nonzero centers sharing a distance offset, a low-width quadratic approximation, and necessary budgets on IMQ coefficients.
Critical strengths
-
This kernal theory paper features a detailed controlled covariance comparison, as the authors summarized in the second paragraph of section 6. The calculation and its moment justification look sound, making this the strongest contribution of this paper.
-
The paper also provided specific representation analysis.
Critical weaknesses
-
The theoretical advance is not deep and not well positioned against prior work. The basic zero-bias Yat construction predates the submission, and several guarantees are standard kernel arguments; the covariance and representation results are useful but narrow. Three-atom minimality concerns exact equality with a shared distance offset, whereas one growing-bias atom can uniformly approximate an IMQ section. The paper should clarify its specific new increment and cite the earlier construction.
-
The approximation comparison leaves the cost tradeoff unresolved: Yat bounds unit count while center lengths and kernel norms grow, whereas IMQ bounds total absolute coefficients, so comparable function-norm efficiency is not established. The authors correctly claim neither a matching width lower bound nor improved generalization; experiments add little weight given unmatched tuning, mixed zero-shot gains, and unverifiable records.
-
The paper is hard to read, with many nonations weren't properly defined before introduction.
Questions for authors
-
Is there any further practical motivation of developing this Yat kernal theory, beyond answering the pure mathematical question of "when we add a directional component to a distance-based kernel, what properites change and what are preserved", which is a reasonable mathematical question by itself?
-
How does the construction relate to arXiv:2411.08085? Please cite and compare the earlier zero-bias operation and neuron use, and identify what is genuinely new in the current work.
-
Can the language-model experimental results be verified or reproduced?
-
Is there a concrete finite-error setting in which the paper’s coefficient thresholds and Yat construction yield a useful tradeoff once center lengths and kernel norms are taken into account?
Thank you for the careful reading. We address your concerns below.
1. Practical motivation. Along a ray with nonzero alignment, ⵟ’s response retains sensitivity to its tied center. Proposition 7 gives a nonzero tied-center gradient limit for ⵟ, while IMQ's center gradient decays as Θ(r−3). The ⵟ input gradient still decays as O(r−1). Unit derivatives do not guarantee loss gradients. Amirian and Schwenker discuss attenuated RBF gradients for inputs initially far from centers. Amirian and Schwenker, §IV
ⵟ FFNs trained in all 12 blocks of a 261M Transformer on 5.22B C4 tokens. Shared bias with learned scale gave 39.07 ± 0.39 WikiText perplexity versus GELU’s 44.23 ± 1.44 (three seeds, mean ± SD), at 655K versus 715K tokens/s (Table 4). Zero-shot results were mixed (Table 5). Our exploratory IMQ run at that scale collapsed (Appendix F.4). Equal tuning budgets and the center gradient’s role remain open.
On the displaced disk, moving the distance center with alignment fixed lowered relative MSE from 80.96% to 66.16%. The best fit, 29.65%, used a separate 2D center. The radial Gaussian reached 51.15%, ahead of both tied fits (52.26% centered, 56.42% Eq. 1) (§3.4, Appendix F.5, Table 6).
2. What changed from earlier work? This study grew out of the 2024 preprint’s zero-numerator-bias ⵟ neuron, with a learned scale and a separate output bias. The 2025 preprint established zero-bias positive semidefiniteness and the single-atom directional limit. We discuss both in §1 and Appendix A.3. Bouhsine (2024), §3.1 · Bouhsine (2025), v2, Apps. G.2–G.3
At b=0, we have universality on compacts excluding zero. On compacts containing zero, the uniform RKHS closure consists of continuous functions vanishing there. An output bias restores density by approximating f−f(0) and adding f(0), leaving the kernel unchanged (Proposition 4, Appendix B.2). Here b shifts the inner product before squaring. For b>0, b2hε⪯kb,ε gives IMQ RKHS inclusion and universality on all compacts (§2). Aronszajn, §7 · Micchelli, Xu, and Zhang, Theorem 17
Our additions include strict global RKHS inclusion for b>0, Gaussian and spherical covariance limits, and a finite-span trace with no continuous RKHS extension (§3). At a nonzero center, three variable-bias atoms with a shared offset are necessary for exact IMQ recovery (Theorem 2). For uniform approximation, one scaled atom suffices as its bias grows (Lemma 5).
3. Reproducibility. Code, configs, and tokenizer are in the supplement. GELU seeds 0 1 2. Shared-bias ⵟ seeds 0 1 2. Histories reconstruct all 40 reported cells. From each seed’s first complete pass, WikiText perplexity is 39.06 for learned-scale shared-bias ⵟ and 44.24 for GELU. Historical training and evaluator revisions, scored checkpoints, and long-context data remain unpinned (Appendix F.4).
4. What do the shared budgets buy? In Theorem 3 (§4.2), the shared budget is the sum of absolute coefficients on raw atoms at fixed offset. See Bach, §2.1. For a PSD quadratic with matrix A, ⵟ reaches any accuracy with at most rank(A) atoms at budget tr(A). Every fixed IMQ budget leaves positive uniform error even with unrestricted width and centers. Otherwise Montel gives a bounded holomorphic limit equal to an unbounded quadratic along a line. We leave the divergence rate open. The ⵟ construction uses growing centers and an RKHS norm of order δ−2 at error δ. Matched-width or matched-norm advantages remain unproved.
For a constructed example on 10≤∥x∥≤20, take g=k1,1(w,⋅)/2 for unit w. Cap center radii, raw coefficient variation, and the common ambient H1,1 norm at one. One ⵟ atom fits exactly. The IMQ coefficient cap implies the norm cap by kernel order. At 10w, g(10w)=121/164 and every admissible IMQ fit has magnitude at most 1/82, giving uniform error at least 119/164. This is an existence example under those caps.
5. Readability. We added a same-projection, different-distance example near Eq. (1). §2 and Table 1 collect notation; Appendix A.1 gives definitions. Appendices C.2.1 and D.3 explain when bounds cease to be informative and why. The conclusion explains the ablation and states that its best fit uses a separate center.