acceptodds
Under review as a conference paper at ICLR 2027

Deep Operator-Valued Spectral Kernel Networks

Abstract

Deep Spectral Kernel Networks (DSKNs) construct nonstationary representations via paired-frequency maps, but independent phase sampling collapses cross-frequency interactions in expectation at initialization to a stationary covariance. We show that sharing a single phase per frequency pair restores the exact harmonizable integrand. Building on this, we introduce Deep Operator-Valued Spectral Kernel Networks (DOSKNs), a primal framework that extends deep spectral learning to vector-valued outputs while decoupling spectral resolution, layer width, and subspace rank. Grouped output coupling enforces block-rank constraints, regularizing cross-dimension interactions and lowers parameter count. Furthermore, we introduce a trace-constrained squared group regularizer with an exact proximal map. Empirical evaluations confirm that DOSKN addresses stationary collapse, cuts coupling parameters by up to 76%, and consistently matches or outperforms unconstrained models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Deep Spectral Kernel Learning, Operator-Valued Kernels, Nonstationary Kernels

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_344rOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper builds on Deep Spectral Kernel Networks (DSKNs; Xue et al., 2019), which build nonstationary representations from paired-frequency feature maps. It makes three contributions:

Phase analysis. In the single-coordinate stochastic phase map of DSKN, the two phases of a frequency pair are sampled independently. The paper shows that this makes the cross-frequency terms vanish in expectation, so the expected kernel at initialization is stationary. Sharing one phase per frequency pair recovers the full harmonizable integrand. DOSKN. The paper extends DSKNs to vector-valued outputs. Each spectral component is paired with a positive semidefinite (PSD) output-coupling operator. Spectral components are grouped by frequency-norm quantiles, and each group shares an orthonormal r-dimensional output subspace. This places a block-rank constraint on the output weights. The construction also decouples the number of frequencies from the layer width. Regularizer. Optimizing the component amplitudes under a trace budget gives a squared group-norm regularizer Ω, for which the paper derives an exact O(M log M) proximal operator.

Critical strengths

The phase analysis is correct and clearly checked. The derivation that independent phases make the cross-frequency terms vanish in expectation, while a shared phase recovers them, is short and correct (App. A.1–A.2). Table 2 confirms it numerically: with a shared phase, the kernel error decreases as M grows, while with independent phases it stays at a constant level.

The work is reproducible. The code is provided in the supplementary material. The proximal operator is checked numerically against bisection and by random perturbation tests.

Critical weaknesses
  1. Limited novelty, and the comparisons leave out the relevant baselines

Each component builds closely on existing work, and what the authors add on top is small:

The phase result is a two-line trigonometric identity. The grouped low-rank output coupling is a form of the low-rank and coregionalization structure that is standard in multi-output kernel learning (e.g., the Linear Coregionalization Model, which the paper itself reviews in Sec. 2). The squared group-norm regularizer and its link to multiple kernel learning come from Micchelli & Pontil (2005) and Bach (2008), as the authors acknowledge (L283–285). Its proximal operator is computed with a standard sort-and-threshold procedure.

The baselines are an MLP, a stationary random Fourier features (RFF) network, and two variants of the 2019 DSKN. The comparison includes no multi-output or operator-valued kernel method, and no more recent nonstationary or deep kernel approach. This is so even though contribution (5) claims parameter savings "relative to … operator-valued baselines", and the paper discusses Brault et al. (2016), multi-output Gaussian processes (MOGPs), and deep kernel learning in its related work.

  1. Presentation and experimental scale No figures. The paper has no figures at all. The paired-frequency construction, the grouped subspace coupling and the layer structure are all explained only through equations and Table 1. One or two diagrams of the architecture and the coupling geometry would make the paper much easier to follow. Unresolved citation and typos. The citation at L51 is unresolved ("(?)"), "SARCOS" is misspelled as "SACROS" (L407), and App. B.4 contains errors such as "accross" and "while while" (L854–856). Small experiments. The real-data experiments use small datasets: two regression tasks, and three multi-label tasks with as few as 593 samples. The synthetic coupling experiment generates data from exactly the block-rank structure the model assumes (Q* = 4, r* = 3), which favours the proposed method.
Questions for authors
  1. The central correction brings no measurable benefit in the experiments

On SARCOS (Table 5), the original DSKN with independent phases (0.0912 ± 0.0014) is slightly better than the corrected DSKN (0.0916 ± 0.0011) and better than DOSKN (0.0923 ± 0.0002). On Parkinson's Telemonitoring, the corrected DSKN (1.2900) is worse than the original (1.2466).

  1. The parameter savings are much smaller than claimed

The headline savings ("cuts coupling parameters by up to 76%", Abstract; 52.5% on SARCOS, Table 5) count only the output-coupling parameters. Total model size is dominated by the frequency parameters, which number 2·M·D_{l−1} per layer (L248–252). For example, in Table 6 the two-layer model has 2,879 coupling parameters out of 48,831 in total, about 6%. Even a large cut in coupling parameters therefore changes the overall model size very little. The paper should report total parameter counts next to every claimed reduction.

To be honest, I believe the presentation of this work needs to be revised to make it easier for readers to understand. At the very least, I found it extremely difficult to follow. Of course, if I have misunderstood anything during the review process, I am open to your rebuttal and willing to adjust my score.

Reviewer_bkwqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper identifies a flaw in Deep Spectral Kernel Networks: sampling the two phases of each paired-frequency feature map independently makes the expected Gram matrix at initialization collapse to a stationary covariance, eliminating exactly the nonstationary cross-frequency structure that harmonizable kernels are meant to capture. The authors show that sharing a single phase offset per frequency pair restores an unbiased approximation of the harmonizable integrand. On this corrected foundation they propose Deep Operator-Valued Spectral Kernel Networks (DOSKNs): each layer maps scalar paired-frequency features through operator-valued coupling operators constrained to a union of Q group-specific orthonormal subspaces, yielding block-rank coupling that decouples dictionary size M_l, layer width D_l, and subspace rank r_l; the corrected DSKN is recovered as a special case. They further introduce a trace-constrained squared group regularizer with an exact O(M log M) proximal operator. Experiments cover kernel-recovery diagnostics for the phase fix, synthetic block-rank recovery, SARCOS and Parkinson's regression, and three multi-label datasets, against MLP, stationary RFF networks, and both original and corrected DSKNs.

Critical strengths
  1. The stationary-collapse diagnosis of independent-phase sampling in DSKN is, to my knowledge, new and clearly explained, and the shared-phase fix is proven unbiased.

  2. The comparisons include the authors' own "corrected DSKN with free linear mixing," which cleanly separates the benefit of the phase fix from the benefit of the operator-valued coupling

Critical weaknesses
  1. The central architectural claim lacks its most important control baselines. The claim is that spectrally grouped, block-rank output coupling (frequency-norm quantile bins sharing orthonormal bases) is a beneficial inductive bias. Yet every comparison is against unconstrained mixing or degenerate couplings.

  2. On SARCOS, DOSKN (0.0923) is slightly worse than both the original DSKN (0.0912) and the corrected DSKN (0.0916) it builds on. The abstract's "consistently matches or outperforms unconstrained models" is stronger than the tables support.

Questions for authors
  1. Please add generic low-rank factorized mixing baselines (C approximating UV, or per-group factorizations without the orthonormality/quantile structure) at matched coupling-parameter budget on the synthetic task, SARCOS, and Parkinson's, plus a random-assignment-to-groups control. Does spectral quantile grouping genuinely outperform arbitrary low-rank structure, or is the observed effect simply rank regularization?

  2. How do external multi-output baselines perform on these tasks - in particular a shallow operator-valued random-feature model with a comparable feature count, and a multi-output GP (e.g., a linear model of coregionalization)? Both lines are cited as motivation but never compared against.

Reviewer_SaCbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Deep Operator-Valued Spectral Kernel Networks (DOSKNs) for learning nonstationary representations with vector-valued outputs. The authors first identify a limitation in the original DSKN, where independent phase sampling removes cross-frequency interactions in expectation, and address it using a shared-phase design. They then introduce grouped low-rank output coupling to model dependencies across output dimensions while reducing the number of parameters. The paper also provides theoretical analysis of the proposed regularization method and evaluates DOSKN on synthetic and real-world datasets, showing competitive performance with fewer output coupling parameters.

Critical strengths
  1. The paper provides a clear theoretical analysis of the phase sampling issue in the original DSKN. The authors show why independent phase sampling removes cross-frequency interactions in expectation and how sharing the phase can address this issue.
  2. The paper includes a good range of experiments, covering both synthetic and real-world datasets. The authors also provide ablation studies on output coupling, network depth, and regularization, which help explain how different components affect performance. The results show that DOSKN can achieve competitive performance while using fewer output coupling parameters.
Critical weaknesses
  1. The novelty of the proposed method is somewhat limited. The grouped subspace coupling is closely related to existing low-rank factorization techniques, while the proposed regularization also builds on established multiple kernel learning methods. The authors should better clarify what makes DOSKN fundamentally different from these existing approaches.
  2. The experimental comparisons are not comprehensive enough. Most baselines are MLPs, random Fourier feature models, and DSKN variants, while established multi-output kernel learning methods discussed in the related work are not included.
Questions for authors
  1. The authors group spectral components based on their frequency norms at initialization and keep these assignments fixed during training. However, the frequencies are continuously updated. Could the authors explain why these fixed groups are still meaningful after training and whether dynamic or learnable grouping could work better?
  2. The theoretical analysis shows that independent phase sampling removes cross-frequency interactions in expectation at initialization. However, both frequencies and phases are learnable during training, and the corrected DSKN does not consistently outperform the original DSKN in Table 5. Could the authors clarify how much the shared-phase design actually helps after end-to-end training?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.