acceptodds
Under review as a conference paper at ICLR 2027

Differentiable Kernel Methods with Error Certificates for Deep Learning Pipelines

Abstract

Modern deep learning pipelines deliver strong accuracy but little per-prediction error control, whereas classical kernel methods come with a mature approximation theory that does not, by default, scale to or plug into such pipelines. We close that gap with two constructs. Sparse kernels make kernel ridge regression scalable: a localized lazy variant that defers computation to inference and solves only small local systems. Differentiable kernel layers make it modular: the same regressor exposed as an end-to-end trainable module, in a doubly-projected form that admits distinct embeddings for the query and the centers while keeping a symmetric positive-definite kernel, and whose three parameter sets (representations, targets, evaluation points) may each be fixed or learned. The two compose, and the resulting design space is broader than end-to-end training alone: training-free transfer, nonlinear probing, hybrid and purely kernel models, classification and policy heads. Our thesis is that these constructs let a deep model carry a computable, geometry-aware error indicator, evaluated at each query and propagated through the pipeline, a kernel-specialized estimate that conformal prediction can build on and a concrete step towards auditable predictions. Across vision, reinforcement learning, out-of-distribution detection and tabular data, such components integrate at competitive cost.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
deep kernel learning, sparse kernel methods, differentiable kernel layers, error certificates, uncertainty quantification, conformal prediction, representation probing, out-of-distribution detection, kernel ridge regression

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_FDrFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a kernel-like layer that can be plugged into modern deep learning pipelines. This kernel-like layer carries with it pointwise kernel theory, offering per-prediction insights of the model. While kernels are traditionally computationally infeasible to implement due to needing a N×NN\times N matrix inversion, the paper introduces sparse kernels that require a square matrix of size M≤NM \le N inversion.

Extensive theory is provided to guarantee the kernel layer, and in particular the sparse kernel, carries the kernel theory needed for per-prediction error bounds. Models with the kernel layer are tested on various datasets, including a transformer and VGG-style convolutional network on image datasets.

Critical strengths

None

Critical weaknesses
  1. There is weak-to-no evidence for the paper’s central claim. In Figure 1, the novel layer achieves lower test accuracy than a standard 2-layer MLP at all labeled budgets (unsure what that term means). In Figure 2, the kernel layers do not appear to improve accuracy above noise for 50k samples (only one run was done so it’s difficult to tell), and for the 1k sample trials, the kernel layer only gets well above per-trial variance on one setup with a vision transformer, and only then, the run extends to a maximum training epoch of 50. Figures 3 and 4 are largely similar: the regular network has performance commensurate with the modified network with the kernel layer. I’ll also note that the novel differentiable kernel layer is not even well indicated (in Figure 1, the trained model with the kernel layer is designated as “SK,” not to mention the legend is in what must be a 3 or 4 fontsize). The only figure that even addresses the central problem raised in the abstract/intro of “per-prediction error control” is Figure 5, although to note, the results are underspecified. The main text notes a $\Delta $ AUROC of +0.123+0.123, although this one quantity is meaningless. What’s the comparison point of the AUROC? What is the relative change rather than absolute? Does this positive change hold across many different settings? Lastly, on the theory side, the MM closest points need to be found heuristically: using the sparse readout while the backbone trains requires re-embedding the full training set at each update, and the main text doesn’t address this cost. To find the distances between neighboring points, one would need to brute-force search the entire training dataset to find nearest neighbors. For practical deep learning systems trained on a minimum of millions of datapoints, this simply cannot be used.

  2. The other major weakness is the readability of everything. It’s overall just unclear what this paper is trying to do: the first five pages read as something theory heavy although this seems to be a theory-motivated methods paper? If it’s a methods paper, why is the majority of the paper definitions of functional forms, spaces, discussion of different bases, tessellations of the input space, error bounds, and remarks? If it’s a theory paper, then why is the abstract, intro, and second half suggesting that this is an input-able layer of deep learning systems? Why is the loss function being defined on page 6? Even accounting for the seemingly double classification, it’s difficult to read on the theory side which is my area of expertise. Many terms are undefined for major portions of the paper (e.g. “doubly-projected form” is mentioned in the abstract and intro but only defined halfway through), have overlap with existing work (e.g. in the first footnote, “lazy” gets redefined from its standard theory definition of a network that cannot feature learn), are overly vague or incorrect (e.g. “the readout is a metric object” specified in the abstract is just incorrect, and it took me a while to figure out the authors were referring to the kernel itself, which acts as a distance measure), are filled with jargon (e.g. “a kernel-specialized estimate that conformal prediction can build on” is incomprehensible to me, maybe someone who’s more familiar with methods literature could parse it). Additionally symbols get overloaded (see yk,θ,θz,θx,θyl(⋅)y^l_{k,\theta,\theta_z,\theta_x,\theta_y}(\cdot) in Eq. (9)) and it’s hard to keep track of the shapes of all of the objects.

Questions for authors
  1. Is this paper meant to be theory or methodology? I would consider raising my score if (a) the work is intended to be theory, and (b) the main body of the paper is drastically changed to reflect the theory focus. Currently, it reads as two half papers, one with weak motivation (theory) and one with weak support (methods), and it appears that the theory work can be more easily built from.
  2. This paper uses kernel layers to have pointwise guarantees of deep learning models. How does this compare to the recursive feature machine of Radhakrishnan, that I will mention is a related work that does not get discussed [1]?
  3. How do the scores shown in Figure 5 compare against the model’s error on said datapoints? It’s unclear if these two are distinct or if the scores shown are just a proxy for the model’s error.

Minor: “desambiguated” typo on line 141

[1] Radhakrishnan, Adityanarayanan, et al. “Mechanism for feature learning in neural networks and backpropagation-free machine learning models.” Science. 2024.

5 comments
AuthorsOpenReview ↗

Response to Reviewer FDrF (i)

We thank the reviewer for the review. We answer the points in order; several rest on details that the submission left implicit, and the revised PDF now states them (pdfdiff against the submission).

The central claim, and the evidence for it. The central claim of the paper is that kernel methods, with the pointwise error estimates of RKHS theory, can serve as components of modern deep learning pipelines at competitive accuracy and cost. To our knowledge, this had not been shown experimentally across such a range of applications. The paper makes small but necessary extensions to existing RKHS theory, and supports them with a wide range of experiments. These required a substantial engineering effort. No existing library provided what they need: a lazy sparse kernel layer, with the per-query power function and gradients with respect to the queries, the centers and the coefficients, usable inside a deep learning training loop. GPyTorch and similar libraries, built around Gaussian-process inference, do not provide this layer. The implementation used here was therefore built on the kernel library of LeFloch et al. (2025) and interfaced with the deep learning backend (Section 2.3). The evidence is of two kinds.

  • Parity at competitive cost. A readout that fits nothing matches trained heads on frozen features (Figures 1 and 2); a purely geometric objective, with no likelihood, trains a backbone (Figure 3); kernel modules replace or augment the heads of two reinforcement-learning agents (Figure 4); and kernel constructions match an MLP and tree ensembles on tabular data (App. A.15).
  • The error indicator. It detects out-of-distribution prompts (Figure 5); a split-conformal wrapper turns it into a detector with a controlled false-positive rate (Table 15); it ranks the readout's errors (AUROC 0.69 to 0.86, App. A.7); and both terms of its composition along a pipeline are now measured on a trained model (new App. A.11.2).

Comparable accuracy in Figures 1 to 4 is therefore the result we report, not a failure to improve: what the kernel components add is that they reach it while carrying the indicator.

Figure 1, and the claim of parity. The reviewer is right about the figure: it plots the readout at M=100M = 100, which trails the MLP at every budget, by 0.5 to 1.4 points (Table 3), while the text quotes M=500M = 500. The revised Figure 1 shows both. At M=500M = 500, the readout is above the MLP at all seven budgets, by 0.1 to 1.3 points (Table 3), without fitting any coefficient, against an MLP whose learning rate was selected for it. The cost is prediction time: 39.6 s against 9.2 s at full budget (Table 2). The "labeled budget" NxN_x is the number of labeled training images used, from 500 to 50,000; the axis now says so. "SK" is the sparse lazy readout, which fits nothing; the legend is now readable and names it.

Figure 2, and noise. At Nx=1000N_x = 1000 the min–max bands overlap, so we read the comparison paired by seed: over the last twenty epochs, the readout is ahead of the model's own head in 82 of 84 paired comparisons on the transformer and 84 of 84 on the convolutional network (App. A.7). At Nx=50,000N_x = 50{,}000 there is one seed, and we make no claim beyond parity there: a readout durably beating a head trained end-to-end for the same objective would point to an undertrained head. All backbones are trained for 50 epochs with a cosine schedule (App. A.7). The point of the figure is not that the readout beats the head, but that it reads the representation at every epoch without fitting anything, and carries a per-query error indicator that the head does not have.

Figures 3 and 4. Figure 3 trains a backbone with a purely geometric objective, with no likelihood and no logits: it beats cross-entropy by 5.5 points on the transformer (same sign on 4 of 4 seeds) and loses 2.5 points on the convolutional network (0 of 4 seeds), and we report both. Figure 4 replaces both heads of a PPO agent at 0.28 times their parameters and reaches the baseline reward; the Double DQN agent with kernel modules reaches 246, against 174 for the baseline and 206 for a parameter-matched control.

AuthorsOpenReview ↗

Response to Reviewer FDrF (ii)

Response (2/2) to Reviewer FDrF

Figure 5: comparison point, relative change, robustness.

  • Comparison point: the Relative Mahalanobis Distance (RMD) of Ren et al. (2021; 2023), the strongest input-side score of the published benchmark. Our re-implementation reproduces its published near-OOD AUROC within 0.010 (0.818 against 0.828).
  • Absolute and relative: on near-OOD, our score reaches an AUROC of 0.941 with the whitened map (paired gain +0.123, 95% CI [+0.119, +0.128]) and 0.892 with the diagonal map (+0.073, CI [+0.070, +0.076]). Relative to RMD, this cuts the distance to a perfect detector (1−AUROC1 - \text{AUROC}) from 0.182 to 0.059, by 68%, and to 0.108 with the diagonal map, by 41%. Far-OOD is at 1.000 for all scores.
  • Robustness: unchanged across seven orders of magnitude of the nugget (0.891 to 0.892), across the scale of the contrast, and across thread configurations; a Matérn kernel tuned at its best reaches 0.887. As a deployed detector with a false-positive rate fixed at 5%, it flags 67.4% of near-OOD prompts against 36.4% for RMD (Table 15).
  • Scope: this is one benchmark (PEGASUS-xsum, three OOD sets). The main text now gives the comparison point and the absolute values, not only the paired difference.

Q3. Are the Figure 5 scores distinct from the model's error? Yes. They are computed from the embeddings alone, with no labels and no decoding, and measure how well the in-domain data cover a prompt. They are not a proxy for the quality of the generated summary, which we do not measure. Where the indicator is compared with errors, in the probing readout, it ranks them with an AUROC rising from 0.69 to 0.86 during training (convolutional network; App. A.7).

Nearest-neighbour cost during training. The submission did not say this clearly enough in the main text, and the revision does. In all our training experiments, the kernel layer is used in its dense form: its centers are the minibatch's own features (Section 3.3) or a small set of learned centers (Section 3.4), so no full training set is re-embedded and no neighbour search is run at any update (Section 2.3, App. A.4). The sparse form is used only to read a frozen representation, embedded once. There, the neighbour search is exact brute force by design, so that no reported number depends on approximate neighbours; its cost grows with NN while the local solve does not (Table 6). Beyond the scales measured here, an approximate index is the natural replacement, as in nearest-neighbour OOD detection (Sun et al., 2022). We have not measured millions of points, and the limitations section says so.

Q1. Theory or methodology? Methodology. The kernel layer is the contribution; the theory is what it carries, namely a computable per-query error indicator, and the tools behind it (Schaback bound, Lipschitz propagation) are classical. We understand that the first five pages read otherwise. The revision adds a roadmap paragraph at the end of Section 1, stating that Section 2 defines what the layer computes and what estimate it carries, and that the method is evaluated in Section 3.

Terminology and notation.

  • "Doubly-projected" is now defined at its first use in the introduction.
  • "Lazy": we use it for deferred computation, and footnote 1 already separates it from the lazy training regime; the revision moves that distinction into the text.
  • "The readout is a metric object": reworded. The intended meaning is that the readout is built on a kernel, that is, on a similarity between representations, which is what suggests the geometric objective of Section 3.3.
  • "A kernel-specialized estimate that conformal prediction can build on": reworded as "a per-query score that a split-conformal wrapper turns into a detector with a controlled false-positive rate (App. A.10.2)".
  • Notation: App. A.1 now includes a table of symbols with their dimensions, including those of Eq. (9).

Q2. Recursive Feature Machines (Radhakrishnan et al., 2024). Thank you for the reference, now discussed in the related work. RFMs learn a Mahalanobis metric for a kernel machine through the average gradient outer product, without backpropagation. The two lines are complementary: RFMs learn the metric of a kernel predictor, while our layer inserts a kernel predictor into a neural pipeline and carries its error indicator. A metric learned by an RFM could serve as the adapter map of our layer, and the indicator would apply to the resulting predictor.

Minor. The typo on line 141 is corrected.

Reviewer_FDrFOpenReview ↗

Response to Authors.

The response does not engage with the specific points raised in my review, despite stating that it addresses them in order. Responses (i) and (ii) are also identical, which made it difficult to tell what was intended to address what.

On the central claim: establishing that a kernel layer can and should be introduced into modern DL pipelines requires showing (1) that it adds no meaningful cost to training or inference, and (2) that it contributes something nontrivial. On (1), the evidence remains marginal: experiments are limited to CIFAR-10, training dynamics are shown only up to ~150 epochs, and the bottom of Fig. 2 shows a notable accuracy drop for the vision transformer. On (2), there is still no clear evidence that the per-sample error indicator is more informative than what other probes would provide.

I was also unable to find the updated PDF. After refreshing and clearing my cache, Figure 1 still shows the legend “SK,” so I cannot evaluate changes that are not visible to me.

The engineering effort behind the implementation is not a factor in my assessment. I recognize the time invested, but the score has to reflect the significance and support of the contribution itself.

My assessment is unchanged, and I am keeping my score.

Reviewer_hL6fOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper discusses a dilemma: Deep networks are accurate but say nothing about how wrong any single prediction might be; kernel methods have pointwise error theory but do not scale and do not plug into trainable pipelines. Therefore, it uses sparse kernels and designs differentiable kernel layers to build a new neural network. The key point is that classical RKHS error bounds ϵk(z)\epsilon_k(z) survive this transfer and can be composed through a deep pipeline (Proposition 1), where conformal wrappers can build on them. Experiments show that lazy sparse readouts match trained heads on frozen features at almost zero fitting cost.

Critical strengths
  1. The paper explains clearly how to balance "provable" and "trainable". It uses the Lagrange basis to make the coefficients function values, which is exactly what lets local cells share a global table and inherit the error bound cell-by-cell. This is an interesting point.

  2. The experimental results are diverse, including different classes. The OOD experiment shows that the indicator is assessed as a predictor rather than an accessory score

Critical weaknesses
  1. The title-level claim is never tested in its own regime. First, what is emphasized as an error certificate "propagated through the pipeline" is only ever evaluated on an interpolating readout attached to a frozen backbone (A.11.1); the distortion term of Eq. 27 is never evaluated numerically anywhere. Second, the certificate is a static, post-hoc object: Eqs. 4/7 hold per frozen configuration but say nothing about the training trajectory, and the surrogate framework (Eq. 27, Prop. 1) presupposes a well-defined target map, where the assumption is undefined for interior blocks during feature learning, leaving the paper's own middle-layer deployments (DQN modules, A.8.3) outside the framework's reach. Third, the certificate is never shown to enable any downstream decision: its value is demonstrated only as an OOD score inside a contrastive scheme borrowed from Ren et al. The authors' own dictum "what is optimized is not what is measured" reads less like an accepted limitation than like the paper's central unresolved question.

  2. The necessity of the kernel framing is unestablished. Stripped of the kernel narrative, Eqs. 9–10 resemble known neural modules: attention-like aggregation over an M-NN graph with samples as nodes, or the memory / codebook line the paper itself cites (Sukhbaatar et al.; Lample et al.). The kernel narrative does not make the module itself new, and brings no new theory.The leading theoretical claim, Proposition 1, combines the classical Schaback bound with standard Lipschitz propagation, and both are known tools, with the Lipschitz half being exactly what a pure neural analysis would use. The only component without a substitute is ϵk\epsilon_k, yet no experiment tests its decision value in its most natural use

  3. The paper is not positioned against neural-network generalization theory. Norm-based / covering-number bounds (Bartlett et al.; Ma et al.) and per-query distance-aware scores are never discussed as competing frameworks. The target function is assumed to lie in the RKHS, which is a standard assumption in interpolation theory but is never justified here relative to weight-norm alternatives. It is also at its weakest along the training trajectory that the paper's interior-layer deployments live on.

  4. The empirical case is thin and scattered. Everything runs at CIFAR-10 / LunarLander scale across five application areas. At full budget, the M=500 readout beats the MLP by only 0.5 points at 4× its prediction cost (Table 2). The geometric objective is architecture- and budget-dependent (wins on ViT at N=1000, loses on CNN and at full budget). The DQN gain partly reflects added capacity (246 vs. 206 for the parameter-matched MLP), and the tabular benchmark loses to XGBoost. There is no experimental comparison against deep kernel learning (Wilson et al.), deep GPs, or restricted kernel machines (Suykens et al.), and the scalable-kernel related work covers only three libraries, dismissing the sketching/preconditioning line (e.g. EigenPro).

  5. Presentation is unsatisfying. The paper makes heavy conceptual demands on the reader without offering intuition even it is cheap. The power function is mentioned Section 2 as the central object but is never given any interpretation beyond its formula — for readers outside scattered-data approximation, the name itself (which has nothing to do with "power") is a barrier, and one sentence ("ϵk\epsilon_k is the worst-case interpolation error over the RKHS unit ball, i.e., the GP posterior standard deviation") would fix it. The M-NN construction, which is essentially "k-nearest neighbours carve the space into cells," is written as an abstract tessellation definition, while a single 2-D figure with M=2 would convey it instantly. Eqs. 4 and 7 state the same bound twice (the second is the first applied cell-wise). The dense notation convention, lowercase xx denoting an N×DN \times D matrix with rows xnx^n, silently reverses the statistics/ML convention and forces constant re-reading. Finally, the three regimes the paper moves among: certificate, error indicator, and score, are used interchangeably in the main text even though the paper's own empirical evidence supports only the weakest ("score"); the main text never states this hierarchy explicitly. These are individually small, but jointly they mean the paper's actual contribution is much smaller than the reading effort it demands.

Questions for authors
  1. The contributions of this paper must be clearly claimed without defensive writing. What is the core theory under your kernel narrative? What makes it different from the current deep kernel methods? And what practical implications can be obtained from this theory?

  2. Please compute, on at least one trained hybrid model (the Double DQN modules, the PPO heads, or the tabular readout), the two terms of (27) at a sample of queries: the distortion ∣M(z)−R(z)∣|M(z) − R(z)| and the Schaback term ϵk(zB,x0A)C\epsilon_k(zB, x_0 A)C, with CC an empirical stand-in of the kind used in Appendix A.11.1. If you can, also report a single number for the Lipschitz product Λ>ℓ\Lambda_{>\ell} on the augmented ViT of Appendix A.8.3, or state that this model is not of kernel type because of the row-wise softmax, and remove it from the scope of Proposition 1 in the main text. If the answer is that no such measurement exists, I will ask that “error theory in deep pipelines” be removed from the list of main contributions or rewritten as “a surrogate composition, untested numerically.”

  3. The paper claims ϵk\epsilon_k as the key deliverable, but it is never tested in its most natural use. Please consider the practical application where ϵk\epsilon_k could be the key indicator, and how this value be interpreted.

4 comments
AuthorsOpenReview ↗

Response (1/3) to Reviewer hL6f

We thank the reviewer for the most detailed review of this submission, and for Q2, which turns the main objection into a concrete test. We answer the three questions first, then the remaining weaknesses. All changes are in the revised PDF (pdfdiff against the submission).

Q1 and the first weakness. What is new, how it differs from deep kernel methods, and what it is for.

What is new. Not the tools. The Schaback bound and Lipschitz propagation are classical, Proposition 1 combines them, and sparse local constructions go back to Shepard and moving-least-squares methods (Haas, 1990; Lazzaro and Montefusco, 2002; Fenn and Steidl, 2006). The thesis of the paper is that kernel methods, with the pointwise error estimates of RKHS theory, can serve as components of modern deep learning pipelines at competitive accuracy and cost. To our knowledge, this had not been shown experimentally across such a range of applications, and the paper is built to establish it: we would like the community to recognize kernel methods as competitive components of deep learning. The paper makes small but necessary extensions to existing RKHS theory, and supports them with a wide range of experiments.

Difference from deep kernel learning, and the engineering it required. Deep kernel learning (Wilson et al., 2016) places a Gaussian process on learned features and trains it through the marginal likelihood or a variational bound. In the exact case, our indicator is that GP's posterior standard deviation (verified to 5⋅10−155\cdot10^{-15}, App. A.10). Our addition is the lazy sparse layer, and what it changes is where the indicator remains available: in a readout that fits nothing, and composed along a pipeline (Proposition 1). No existing library provided this. GPyTorch and similar libraries treat a Gaussian process as a probabilistic model, trained through a variational objective, and their layers return distributions, including in deep GPs. Our experiments need a deterministic kernel layer that any loss can train (cross-entropy, a policy-gradient objective, a geometric objective), that sits anywhere in a pipeline, as a head or alongside a feed-forward block, and that carries its per-query power function, in a lazy sparse form, with gradients with respect to the queries, the centers and the coefficients. The implementation used here was therefore built on the kernel library of LeFloch et al. (2025) and interfaced with the deep learning backend (Section 2.3), which required a substantial engineering effort.

The certificate and its uses (first weakness, third point). Section 3.5 gives first evidence of its value: used as a detector on language-model embeddings, the power function outperforms the strongest input score of the reference benchmark, and a split-conformal wrapper turns it into a detector with a controlled false-positive rate (App. A.10.2; see Q3). Further uses, such as localization and abstention, are studied in a companion paper in progress, as they would not fit in ten pages alongside the thesis of this one. The static nature of the certificate (second point) is answered under W1 below.

AuthorsOpenReview ↗

Response (2/3) to Reviewer hL6f

Q2. The two terms of (27) on a trained hybrid model.

We ran it on a tabular kernel readout: the ARD variant of App. A.15 (learned diagonal metric, learned ridge, dense solve), on the "credit" dataset (OpenML 44089). It is a single-block case (K={L}K=\{L\}), so (30) reduces to (27). Since the readout is a ridge regressor, MM is not the interpolant RR, so the distortion term is not zero by construction. The kernel component is fitted on the residuals of a linear trend, which play the role of the target ff. C^\hat C is the empirical stand-in of App. A.11.1, now measured on one half of the test set and applied to the other, so that the bound is not calibrated on the queries it evaluates.

Three seeds, 2,350 evaluation queries each, mean over seeds:

median90th pct.max
distortion ∣M(z)−R(z)∣\lvert M(z)-R(z)\rvert0.1060.3050.769
observed error ∣f(z)−M(z)∣\lvert f(z)-M(z)\rvert0.2410.6871.46

With C^\hat C at the 95th percentile of the calibration half, the median bound is 1.19 (coverage 98.6%, median ratio of bound to observed error 5.0). With C^\hat C at the maximum, the median bound is 2.20 and 2.29 on two seeds, and 32.8 on the third, where a single query almost coincides with a training point of opposite label (coverage at least 99.96%).

The answer is therefore twofold. The distortion term, the deep-versus-dense discrepancy the reviewer asked about, is small: its median (0.106) is below the observed error (0.241). The bound is nonetheless vacuous on this model, and the Schaback term is the reason: on noisy binary labels, the stand-in C^\hat C for the target norm is large, driven by near-coincident points with opposite labels. The pointwise form also matters: the uniform version of the distortion, ∥M−R∥Hk\|M-R\|_{\mathcal H_k}, lies between 303 and 2,780, against at most 0.78 pointwise.

We therefore accept the reviewer's rewording, updated by this measurement. The contribution, now titled "Error estimates in deep pipelines", reads: "The tools are classical (the Schaback bound and Lipschitz propagation); what is new is the object they apply to. We transfer them to the sparse and deep readouts and compose them along a pipeline through a computable surrogate (Proposition 1), an a posteriori statement about a trained model. On one tabular model its distortion term is measured small and its Schaback term vacuous on noisy binary labels (Appendix A.11.2)." The protocol and the per-seed results are in the new App. A.11.2.

On the augmented ViT of App. A.8.3, the reviewer is right, and our own text was inconsistent. Its module normalizes each kernel row by a softmax: it is a normalized smoother, not an element of Hk\mathcal H_k, as A.8.3 itself states. The "Consequence" paragraph of A.11, which listed this model as of kernel type, was wrong. We removed it from the scope of Proposition 1, in the main text and in A.11.

Q3. The indicator in its most natural use, and how to interpret it.

Interpretation. ϵk(z,x)\epsilon_k(z,x) is the worst-case interpolation error over the unit ball of Hk\mathcal H_k, that is, the GP posterior standard deviation. It depends on the query and the centers only, not on the target. It measures how well the data cover the query, not how hard the label is. Its natural use is to decide when a query lies outside what the training data support.

Evidence of decision value, in increasing order of strength:

  1. Error ranking. On the probing readout, its AUROC for ranking errors rises from 0.69 to 0.86 during training on the convolutional network (App. A.7).
  2. A deployed detector. Wrapped in split-conformal calibration on in-domain data only, its false-positive rate matches the nominal level (0.0092 / 0.0493 / 0.0982 at 0.01 / 0.05 / 0.10). At 5%, it flags 67.4% of near-OOD prompts, against 36.4% for RMD (App. A.10.2, Table 15).
  3. Selective classification (companion study, Anonymous 2026, work in progress, not yet public; CIFAR-10, with CIFAR-100 as near-OOD, three seeds). On a pure in-distribution stream, MSP is the better selector (paired ΔAURC +0.001, CI [+0.001, +0.002]). On a stream contaminated 1:1 with near-OOD, the sign flips and the indicator is the better selector (paired ΔAURC −0.002, CI [−0.003, −0.002]).

This is also our answer to "how to interpret it": the indicator is not a better sorter of in-distribution errors. Its value is in abstention under distribution shift.

AuthorsOpenReview ↗

Response (3/3) to Reviewer hL6f

W1, second point. A static certificate.

Agreed. Eqs. (4), (7) and (27) hold for a frozen configuration. Proposition 1 is an a posteriori statement about a trained model, not about the training trajectory. For interior modules (Double DQN, App. A.9), it applies to the trained model once training is over, and says nothing during feature learning. Section 2 and the contribution list now state this scope. We also agree that "what is optimized is not what is measured" is an open question of the paper rather than a settled limitation; the conclusion now says so.

W2. Necessity of the kernel framing. See Q1: we do not claim new tools. The memory and attention modules the reviewer mentions normalize their weights; App. A.8.3 is precisely the case where this normalization takes the module out of Hk\mathcal H_k and the bound no longer applies as stated.

W3. Positioning against generalization theory.

We read the two references as Bartlett, Foster and Telgarsky (2017) and E, Ma and Wu (2022); please correct us if other works were meant. Neither is a competing framework for what the paper measures. Bartlett et al. bound the misclassification probability, uniformly over networks of bounded spectral complexity. E, Ma and Wu bound the Rademacher complexity of balls of the Barron norm, and their approximation results hold in L2L^2. Both control averaged quantities over a function class. The power function bounds the error at one query, a posteriori and without the target, given the centers, up to the unknown scale ∥f∥Hk\|f\|_{\mathcal H_k}. The two answer different questions; Section 1 now says so.

On the RKHS assumption relative to weight-norm alternatives: the alternative is itself built from RKHSs, since E, Ma and Wu show that the Barron space is a union of reproducing kernel Hilbert spaces (their Proposition 3). The question is therefore less RKHS versus weight norms than which kernel, and the paper does not assume a value of ∥f∥Hk\|f\|_{\mathcal H_k}: App. A.11.1 measures the smallest constant that makes the bound hold at each query: median 1.19 (ViT) and 1.08 (CNN), maximum 2.66 and 2.51, on one task, one depth and one seed.

The distance-aware score closest to ours, SNGP (Liu et al., 2020), is now discussed in the main text: it is a random-feature approximation of the same quantity, which costs 0.042 AUROC on near-OOD (App. A.10).

W4. Empirical case.

We agree with the facts listed, and the paper reports them as such: the M = 500 readout costs 4.3 times the MLP's prediction time for +0.5 points (Table 2), and the DQN gain is partly capacity (parameter-matched control at 206, against 246 and 174, App. A.9).

On tabular data, the purpose of the Deep KRR test (App. A.15) is to show that a kernel construction can replicate a neural architecture: it matches the two-layer MLP (AUC 0.819 vs 0.822). Kernel methods can equally replicate tree ensembles. A kernel variant with a learned diagonal (ARD) metric and a learned ridge, fitted on all labels (by Nyström on the larger dataset), reaches an AUC of 0.8548 on Give Me Some Credit, against 0.8556 for XGBoost, and 0.8527 on the "credit" dataset of the tabular benchmark of Grinsztajn et al. (OpenML 44089), against 0.8528 for a random forest and 0.8517 for XGBoost, with untuned boosting baselines. The revised App. A.15 reports it. Both constructions carry the per-query indicator at an accuracy comparable to these baselines.

Scale is the main limitation and is stated as such. We did not run deep kernel learning, deep GPs, RKM or EigenPro, and we make no claim of outperforming them.

W5. Presentation. We took the five points:

  • Section 2.1 now carries the reviewer's sentence: ϵk\epsilon_k is the worst-case interpolation error over the unit ball of Hk\mathcal H_k, that is, the GP posterior standard deviation.
  • Appendix A.2 now draws the M-NN cells in two dimensions for M = 2, and Section 2.2.1 defines a cell in one sentence before the formal definition.
  • Eq. (7) is now introduced as (4) applied cell by cell, rather than restated.
  • The matrix convention (x∈RN×Dx \in \mathbb R^{N\times D} with rows xnx^n) is announced at first use.
  • Section 2.1 now separates three levels: a certificate (a bound, valid if f∈Hkf\in\mathcal H_k with known norm), an indicator (the computable ϵk\epsilon_k), and a score (its use for ranking). Our evidence supports the indicator and score levels; the certificate level holds only under the assumption measured in Apps. A.11.1 and A.11.2.

Thank you for your detailed responses. My concerns were resolved partially. I did not find effective revisions in the current manuscript. The paper is still uneasy to understand now. I acknowledge the author's effort in investigating the theory about kernel methods, but the current version did not reach the publication standards. I keep my score and encourage the authors to improve readability and make it clear for most readers.

Reviewer_bS2QOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes using sparse kernel methods in place of more standard linear/MLP probes. This has some benefits, as it enables the use of kernel-related machinery to compute uncertainty estimates, and also allows for a sensible initialization by intelligently placing the centres.

Critical strengths

Using 'lazy' kernels for probes is very elegant, as with this kernel we get cheap computation without an approximation.

Critical weaknesses

Generally the paper is unclear, and I am unconvinced by the experiments. In no particular order:

  • In the introduction, linear probes are framed as weak. There is little empirical evidence to motivate this in the paper, and indeed in general I expect a linear probe in front of a strong neural network to be performant.
  • It's unclear what the Figure 3 experiment are? Figure 1 shows that the paper's method is uniformly worse than MLP (and also conflicts with the text)
  • It's unclear what dataset is being used in Figure 3.
  • Some figures are squashed/have squashed legends.
  • Hyperparameters are not tuned for the majority of experiments.

Exposition is overly complex. For example:

  • a cell is defined in a convoluted way, but a cell is just a region in space which shares the same nearest inducing points.

  • sparse Lagrange functions give us the interesting property that the 'global object is the local predictor' (I would lead with this), but we are left unsure of what they are. You mention that the Lagrange function definitions are in the Appendix and then motivate them by mentioning the Lagrange property, but this definition is also in the appendix.

Questions for authors

Please see the Critical weaknesses section

1 comment
AuthorsOpenReview ↗

Response to Reviewer bS2Q

We thank the reviewer for the review, and for finding the lazy construction elegant: cheap computation with no approximation is indeed what it is designed for. We answer the points in order; the revised PDF carries the changes (pdfdiff against the submission).

1. Linear probes framed as weak. We agree, and our own measurements support the reviewer. Against a cross-validated ridge on the same features, the kernel readout gains 1.6 points on the transformer and 0.5 on the convolutional network at Nx=1000N_x = 1000 (Table 9). At full budget, on the convolutional network at depth four, every readout, linear included, lands within three thousandths of the others (App. A.7): cross-entropy has driven that representation into ten near-orthogonal clusters (99.8% mode purity), where any readout suffices. The transformer keeps a continuum (83% purity), and that is where a kernel readout still pays in accuracy. What it adds in every case is a per-query error indicator, which a linear probe does not carry. The introduction now says this instead of framing linear probes as weak.

2. Figure 1 against the text. The reviewer is right: Figure 1 plots the readout at M=100M = 100, which trails the MLP at every budget, by 0.5 to 1.4 points (Table 3), while the text quotes M=500M = 500. The revised Figure 1 shows both. At M=500M = 500, the readout is above the MLP at all seven budgets, by 0.1 to 1.3 points, without fitting any coefficient, at 4.3 times the MLP's prediction time at full budget (Table 2).

3. The Figure 3 experiment. Each backbone is trained from initialization by the geometric objective (17) alone, with no cross-entropy term, and read by the kernel readout at blocks 3 and 4 (blue, red). The black curve is the same backbone trained by cross-entropy with its own head. Architectures, data and optimizer are those of Figure 2, except the duration: 150 epochs at Nx=1000N_x = 1000, where the objective converges more slowly than cross-entropy (App. A.8.1). The caption now states this.

4. The dataset of Figure 3. CIFAR-10, with the two backbones of Section 3.2. Section 3.3 did not say so; the revision states it in the text and in the caption.

5. Squashed figures and legends. Agreed. Figure 1 is redrawn in the revision, and the other figures will be redrawn at the same font size in the final version.

6. Hyperparameters. Two cases. The kernel readout is untuned by design, and its sensitivity is measured: a fourfold sweep of the lengthscale moves accuracy by at most 0.79 points (App. A.6); the nugget, from 10−910^{-9} to 10−310^{-3}, changes accuracy by at most 0.001 (App. A.7); the bandwidth MM is swept in Tables 3 and 8. The baselines are not untuned: the learning rate of the MLP head was selected for it, which App. A.6 now states. Our claim is therefore parity without fitting, against tuned neural heads, not superiority.

Exposition.

  • Cells. A cell is the set of points that share the same MM nearest centers; Section 2.2.1 now says so in one sentence before the formal definition.
  • Sparse Lagrange functions. We follow the suggestion: Section 2.2.2 now opens on the property that on each cell the global object is the local predictor, and the closed form (11) with its Lagrange property moves from the appendix into the main text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.