- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a kernel-like layer that can be plugged into modern deep learning pipelines. This kernel-like layer carries with it pointwise kernel theory, offering per-prediction insights of the model. While kernels are traditionally computationally infeasible to implement due to needing a matrix inversion, the paper introduces sparse kernels that require a square matrix of size inversion.
Extensive theory is provided to guarantee the kernel layer, and in particular the sparse kernel, carries the kernel theory needed for per-prediction error bounds. Models with the kernel layer are tested on various datasets, including a transformer and VGG-style convolutional network on image datasets.
Critical strengths
None
Critical weaknesses
-
There is weak-to-no evidence for the paper’s central claim. In Figure 1, the novel layer achieves lower test accuracy than a standard 2-layer MLP at all labeled budgets (unsure what that term means). In Figure 2, the kernel layers do not appear to improve accuracy above noise for 50k samples (only one run was done so it’s difficult to tell), and for the 1k sample trials, the kernel layer only gets well above per-trial variance on one setup with a vision transformer, and only then, the run extends to a maximum training epoch of 50. Figures 3 and 4 are largely similar: the regular network has performance commensurate with the modified network with the kernel layer. I’ll also note that the novel differentiable kernel layer is not even well indicated (in Figure 1, the trained model with the kernel layer is designated as “SK,” not to mention the legend is in what must be a 3 or 4 fontsize). The only figure that even addresses the central problem raised in the abstract/intro of “per-prediction error control” is Figure 5, although to note, the results are underspecified. The main text notes a $\Delta $ AUROC of , although this one quantity is meaningless. What’s the comparison point of the AUROC? What is the relative change rather than absolute? Does this positive change hold across many different settings? Lastly, on the theory side, the closest points need to be found heuristically: using the sparse readout while the backbone trains requires re-embedding the full training set at each update, and the main text doesn’t address this cost. To find the distances between neighboring points, one would need to brute-force search the entire training dataset to find nearest neighbors. For practical deep learning systems trained on a minimum of millions of datapoints, this simply cannot be used.
-
The other major weakness is the readability of everything. It’s overall just unclear what this paper is trying to do: the first five pages read as something theory heavy although this seems to be a theory-motivated methods paper? If it’s a methods paper, why is the majority of the paper definitions of functional forms, spaces, discussion of different bases, tessellations of the input space, error bounds, and remarks? If it’s a theory paper, then why is the abstract, intro, and second half suggesting that this is an input-able layer of deep learning systems? Why is the loss function being defined on page 6? Even accounting for the seemingly double classification, it’s difficult to read on the theory side which is my area of expertise. Many terms are undefined for major portions of the paper (e.g. “doubly-projected form” is mentioned in the abstract and intro but only defined halfway through), have overlap with existing work (e.g. in the first footnote, “lazy” gets redefined from its standard theory definition of a network that cannot feature learn), are overly vague or incorrect (e.g. “the readout is a metric object” specified in the abstract is just incorrect, and it took me a while to figure out the authors were referring to the kernel itself, which acts as a distance measure), are filled with jargon (e.g. “a kernel-specialized estimate that conformal prediction can build on” is incomprehensible to me, maybe someone who’s more familiar with methods literature could parse it). Additionally symbols get overloaded (see in Eq. (9)) and it’s hard to keep track of the shapes of all of the objects.
Questions for authors
- Is this paper meant to be theory or methodology? I would consider raising my score if (a) the work is intended to be theory, and (b) the main body of the paper is drastically changed to reflect the theory focus. Currently, it reads as two half papers, one with weak motivation (theory) and one with weak support (methods), and it appears that the theory work can be more easily built from.
- This paper uses kernel layers to have pointwise guarantees of deep learning models. How does this compare to the recursive feature machine of Radhakrishnan, that I will mention is a related work that does not get discussed [1]?
- How do the scores shown in Figure 5 compare against the model’s error on said datapoints? It’s unclear if these two are distinct or if the scores shown are just a proxy for the model’s error.
Minor: “desambiguated” typo on line 141
[1] Radhakrishnan, Adityanarayanan, et al. “Mechanism for feature learning in neural networks and backpropagation-free machine learning models.” Science. 2024.
5 comments
Response to Reviewer FDrF (ii)
Response (2/2) to Reviewer FDrF
Figure 5: comparison point, relative change, robustness.
- Comparison point: the Relative Mahalanobis Distance (RMD) of Ren et al. (2021; 2023), the strongest input-side score of the published benchmark. Our re-implementation reproduces its published near-OOD AUROC within 0.010 (0.818 against 0.828).
- Absolute and relative: on near-OOD, our score reaches an AUROC of 0.941 with the whitened map (paired gain +0.123, 95% CI [+0.119, +0.128]) and 0.892 with the diagonal map (+0.073, CI [+0.070, +0.076]). Relative to RMD, this cuts the distance to a perfect detector () from 0.182 to 0.059, by 68%, and to 0.108 with the diagonal map, by 41%. Far-OOD is at 1.000 for all scores.
- Robustness: unchanged across seven orders of magnitude of the nugget (0.891 to 0.892), across the scale of the contrast, and across thread configurations; a Matérn kernel tuned at its best reaches 0.887. As a deployed detector with a false-positive rate fixed at 5%, it flags 67.4% of near-OOD prompts against 36.4% for RMD (Table 15).
- Scope: this is one benchmark (PEGASUS-xsum, three OOD sets). The main text now gives the comparison point and the absolute values, not only the paired difference.
Q3. Are the Figure 5 scores distinct from the model's error? Yes. They are computed from the embeddings alone, with no labels and no decoding, and measure how well the in-domain data cover a prompt. They are not a proxy for the quality of the generated summary, which we do not measure. Where the indicator is compared with errors, in the probing readout, it ranks them with an AUROC rising from 0.69 to 0.86 during training (convolutional network; App. A.7).
Nearest-neighbour cost during training. The submission did not say this clearly enough in the main text, and the revision does. In all our training experiments, the kernel layer is used in its dense form: its centers are the minibatch's own features (Section 3.3) or a small set of learned centers (Section 3.4), so no full training set is re-embedded and no neighbour search is run at any update (Section 2.3, App. A.4). The sparse form is used only to read a frozen representation, embedded once. There, the neighbour search is exact brute force by design, so that no reported number depends on approximate neighbours; its cost grows with while the local solve does not (Table 6). Beyond the scales measured here, an approximate index is the natural replacement, as in nearest-neighbour OOD detection (Sun et al., 2022). We have not measured millions of points, and the limitations section says so.
Q1. Theory or methodology? Methodology. The kernel layer is the contribution; the theory is what it carries, namely a computable per-query error indicator, and the tools behind it (Schaback bound, Lipschitz propagation) are classical. We understand that the first five pages read otherwise. The revision adds a roadmap paragraph at the end of Section 1, stating that Section 2 defines what the layer computes and what estimate it carries, and that the method is evaluated in Section 3.
Terminology and notation.
- "Doubly-projected" is now defined at its first use in the introduction.
- "Lazy": we use it for deferred computation, and footnote 1 already separates it from the lazy training regime; the revision moves that distinction into the text.
- "The readout is a metric object": reworded. The intended meaning is that the readout is built on a kernel, that is, on a similarity between representations, which is what suggests the geometric objective of Section 3.3.
- "A kernel-specialized estimate that conformal prediction can build on": reworded as "a per-query score that a split-conformal wrapper turns into a detector with a controlled false-positive rate (App. A.10.2)".
- Notation: App. A.1 now includes a table of symbols with their dimensions, including those of Eq. (9).
Q2. Recursive Feature Machines (Radhakrishnan et al., 2024). Thank you for the reference, now discussed in the related work. RFMs learn a Mahalanobis metric for a kernel machine through the average gradient outer product, without backpropagation. The two lines are complementary: RFMs learn the metric of a kernel predictor, while our layer inserts a kernel predictor into a neural pipeline and carries its error indicator. A metric learned by an RFM could serve as the adapter map of our layer, and the indicator would apply to the resulting predictor.
Minor. The typo on line 141 is corrected.
Response to Authors.
The response does not engage with the specific points raised in my review, despite stating that it addresses them in order. Responses (i) and (ii) are also identical, which made it difficult to tell what was intended to address what.
On the central claim: establishing that a kernel layer can and should be introduced into modern DL pipelines requires showing (1) that it adds no meaningful cost to training or inference, and (2) that it contributes something nontrivial. On (1), the evidence remains marginal: experiments are limited to CIFAR-10, training dynamics are shown only up to ~150 epochs, and the bottom of Fig. 2 shows a notable accuracy drop for the vision transformer. On (2), there is still no clear evidence that the per-sample error indicator is more informative than what other probes would provide.
I was also unable to find the updated PDF. After refreshing and clearing my cache, Figure 1 still shows the legend “SK,” so I cannot evaluate changes that are not visible to me.
The engineering effort behind the implementation is not a factor in my assessment. I recognize the time invested, but the score has to reflect the significance and support of the contribution itself.
My assessment is unchanged, and I am keeping my score.
Response to Reviewer FDrF (i)
We thank the reviewer for the review. We answer the points in order; several rest on details that the submission left implicit, and the revised PDF now states them (pdfdiff against the submission).
The central claim, and the evidence for it. The central claim of the paper is that kernel methods, with the pointwise error estimates of RKHS theory, can serve as components of modern deep learning pipelines at competitive accuracy and cost. To our knowledge, this had not been shown experimentally across such a range of applications. The paper makes small but necessary extensions to existing RKHS theory, and supports them with a wide range of experiments. These required a substantial engineering effort. No existing library provided what they need: a lazy sparse kernel layer, with the per-query power function and gradients with respect to the queries, the centers and the coefficients, usable inside a deep learning training loop. GPyTorch and similar libraries, built around Gaussian-process inference, do not provide this layer. The implementation used here was therefore built on the kernel library of LeFloch et al. (2025) and interfaced with the deep learning backend (Section 2.3). The evidence is of two kinds.
Comparable accuracy in Figures 1 to 4 is therefore the result we report, not a failure to improve: what the kernel components add is that they reach it while carrying the indicator.
Figure 1, and the claim of parity. The reviewer is right about the figure: it plots the readout at M=100, which trails the MLP at every budget, by 0.5 to 1.4 points (Table 3), while the text quotes M=500. The revised Figure 1 shows both. At M=500, the readout is above the MLP at all seven budgets, by 0.1 to 1.3 points (Table 3), without fitting any coefficient, against an MLP whose learning rate was selected for it. The cost is prediction time: 39.6 s against 9.2 s at full budget (Table 2). The "labeled budget" Nx is the number of labeled training images used, from 500 to 50,000; the axis now says so. "SK" is the sparse lazy readout, which fits nothing; the legend is now readable and names it.
Figure 2, and noise. At Nx=1000 the min–max bands overlap, so we read the comparison paired by seed: over the last twenty epochs, the readout is ahead of the model's own head in 82 of 84 paired comparisons on the transformer and 84 of 84 on the convolutional network (App. A.7). At Nx=50,000 there is one seed, and we make no claim beyond parity there: a readout durably beating a head trained end-to-end for the same objective would point to an undertrained head. All backbones are trained for 50 epochs with a cosine schedule (App. A.7). The point of the figure is not that the readout beats the head, but that it reads the representation at every epoch without fitting anything, and carries a per-query error indicator that the head does not have.
Figures 3 and 4. Figure 3 trains a backbone with a purely geometric objective, with no likelihood and no logits: it beats cross-entropy by 5.5 points on the transformer (same sign on 4 of 4 seeds) and loses 2.5 points on the convolutional network (0 of 4 seeds), and we report both. Figure 4 replaces both heads of a PPO agent at 0.28 times their parameters and reaches the baseline reward; the Double DQN agent with kernel modules reaches 246, against 174 for the baseline and 206 for a parameter-matched control.