acceptodds
Under review as a conference paper at ICLR 2027

Learned Shared Space for Kernel-based Meta-learning

Abstract

Kernel methods are attractive for few-shot regression because squared-loss predictors can be computed in closed form from a small set of observations. Their performance depends on how well the kernel captures inherent structure across tasks, motivating the learning of a kernel representation through meta-learning. However, learning the kernel alone does not provide a learned initial function as in standard gradient-based meta-learning. We develop a kernel meta-learning method, based on inducing points, to learn both a common function space and initial function within it. The learned initial function is then transferred to each new task and adapted by functional gradient descent. We characterize the resulting finite adaptation trajectory to better understand the role of gradient steps during adaptation. We further derive excess transfer risk bounds that quantify the effects of approximation, initialization, and adaptation steps. Under the stated assumptions including boundedness, regularity and spectral decay condition of the kernels, we provide an early-stopping rule and the resulting risk bounds, and describe the trade-off between the approximation error of the learned common space along with the cost of learning the inducing points.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Kernel Meta-learning, Meta-learning, Few-shot learning, Kernel methods, Early stopping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_94JzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies few-shot regression, where a model must adapt to a new task using only a small number of labeled examples. The proposed method learns a shared set of kernel basis functions using a neural feature extractor and inducing points. Different tasks share the same basis functions and use task-specific coefficients. The method first pre-trains the shared representation using a variational objective. During meta-training, it solves a quadratic optimization problem to compute a shared initialization for the coefficients. Each task then adapts from this initialization using gradient descent. The shared representation is further updated based on the query losses. For a new task, the model keeps the shared representation fixed and only updates the task-specific coefficients using a few labeled examples. The paper also analyzes how the number of adaptation steps affects prediction error. It derives error bounds that motivate early stopping rules. Experiments on synthetic and real-world datasets compare the method with MAML, R2-D2, DKT, and iKML. The authors also examine the effects of pre-training and shared initialization through ablation studies.

Critical strengths
  1. The paper provides a structured analysis of functional gradient adaptation within the learned kernel space. The conditional and uniform excess transfer risk bounds help characterize the effects of approximation error, initialization, and adaptation depth. This analysis is a useful aspect of the work.

  2. The experiments cover both synthetic and real-world regression tasks. The ablations in Section 5.4 and Appendix F provide evidence that variational pre-training and the proposed shared initialization can improve predictive performance, particularly on the real-world datasets.

Critical weaknesses
  1. The paper addresses a well-studied few-shot regression setting. Its framework builds on established ideas in kernel learning and gradient-based meta-learning. The main methodological difference appears to be the quadratic objective for shared initialization. However, the paper does not sufficiently explain the advantages of this design over existing approaches.

  2. The evaluation mainly compares against MAML, R2-D2, DKT, and iKML, introduced between 2017 and 2022. These are relevant baselines. However, the paper does not establish whether they are sufficient to assess the proposed method's competitiveness. A broader comparison with more recent or closely related approaches would strengthen the evaluation.

Questions for authors
  1. What is the main advantage of the proposed quadratic shared initialization over existing meta-learning approaches? In particular, how does it compare with directly meta-learning the initial coefficients under the same kernel representation?

  2. Table 1 uses independently sampled real-world test tasks for the proposed method and the baselines. Could the authors clarify why the same test tasks were not used for all methods?

3 comments

We thank the reviewer for the careful reading and assessment. Below we provide the response to the stated weaknesses and questions. We uploaded a revised PDF in which the changes are marked in blue.


W1. The paper addresses a well-studied few-shot regression setting. Its framework builds on established ideas in kernel learning and gradient-based meta-learning. The main methodological difference appears to be the quadratic objective for shared initialization. However, the paper does not sufficiently explain the advantages of this design over existing approaches.

The quadratic objective is one part of a construction for gradient-based adaptation in kernel meta-learning. In existing kernel meta-learning methods, the coefficients are tied to the support inputs. This changes their meaning across tasks so that shared initial function is not well defined. Inducing points provide bases shared by all tasks, and in these coordinates gradient descent on the head is functional gradient descent. Within this space, the quadratic objective gives the initial head in closed form. Its first term, the adaptation energy, measures how far the initial function is located from the functions adapted with few support steps on each task. The quadratic objective reduces it, thereby lowering the risk bound (in Theorem 3).


W2. The evaluation mainly compares against MAML, R2-D2, DKT, and iKML, introduced between 2017 and 2022. These are relevant baselines. However, the paper does not establish whether they are sufficient to assess the proposed method's competitiveness. A broader comparison with more recent or closely related approaches would strengthen the evaluation.

The baselines were chosen by how closely they relate to our setting: gradient-based adaptation and kernel-based meta-learning for few-shot regression. MAML is the standard gradient-based method, and DKT, R2-D2 and iKML are representative kernel-based meta-learning methods for few-shot regression with available implementations. To our knowledge, more recent kernel-based methods for this setting are few. Recent models for few-shot regression, such as ScaML-GP (AISTATS 2024), which adapts by Gaussian-process posterior inference, and TNP (ICML 2022) with its variants, which predicts from the support set with a learned transformer encoder, are not direct counterparts of our method. Instead, we backed our comparison with the MAML. The revised Table 1 now reports MAML trained with one and ten inner steps, each with its test depth also selected by minimizing the validation loss.


Q1. What is the main advantage of the proposed quadratic shared initialization over existing meta-learning approaches? In particular, how does it compare with directly meta-learning the initial coefficients under the same kernel representation?

For a given representation, the head is available in closed form. Its leading term is the adaptation energy, where reducing it lowers the risk bound in Theorem 3. Thus, such choice of the head is tied to the analysis. We do not claim an empirical advantage over a head trained directly. A comparison with such a head, with the representation retrained for each choice is left as future work. We also state it as a limitation in Section 6.


Q2. Table 1 uses independently sampled real-world test tasks for the proposed method and the baselines. Could the authors clarify why the same test tasks were not used for all methods?

We agree that all methods should be evaluated on the same test tasks. We have re-evaluated all methods on the same validation and test tasks on the real-world datasets. In addition, MAML is now reported with E=1E=1 and E=10E=10, and for each the test depth is also selected on these validation tasks over T≤30T\le30. The results are below (revised Table 1; In Appendix, Table 9 gives the full results on the depth selection of MAML). The ranking between baselines is unchanged.

SinusoidalOmegaMisfitOracleStructuredAir qualityMETR-LAPEMS-BAY
Ours, val$(T^\ast)$0.03550.17800.22360.42970.35920.05030.34630.0514
Ours, bound$(T^\ast)$0.04390.17920.34320.44670.36150.05270.38570.0513
MAML (E=1E=1), T=1T=11.08871.46401.25081.55130.73630.05040.38850.0904
MAML (E=1E=1), val$(T^\ast)$0.44540.69430.61480.70050.63780.04220.38850.0652
MAML (E=10E=10), T=10T=100.37760.61150.50881.20060.51400.05140.40470.0846
MAML (E=10E=10), val$(T^\ast)$0.29700.51320.44130.92540.49610.04700.39810.0847
R2-D20.04640.20250.21580.47140.62090.06310.42040.0975
iKML2.87621.41322.95320.29831.33120.07280.69640.1294
DKT0.19370.18921.20800.29590.50550.13540.81040.2639

Thank you for the response and additional experiments. The clarifications address some of my questions, and I appreciate the re-evaluation on the same validation and test tasks and the strengthened MAML comparison. However, the advantage of the proposed quadratic initialization over directly meta-learning the initial coefficients under the same kernel representation remains unclear. My main concern about the methodological contribution therefore remains unresolved, and I maintain my original rating.

Thank you for the follow-up comment and for acknowledging the re-evaluation and the baseline comparison. We understand the remaining concern and will address it in future work, as stated in Section 6. Thank you again for your time and consideration.

Reviewer_mYbyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a kernel meta-learning method for few-shot regression. It uses inducing points to define a shared function space and learns an initial function that is adapted to each new task with a few gradient steps. Training combines sparse variational pre-training with query-loss-based refinement. Under stated assumptions, the theory analyzes approximation error, finite-step adaptation, and early stopping. Experiments on five synthetic task families and three real-world datasets compare the method with existing approaches and examine the effects of pre-training, initialization, and adaptation depth.

Critical strengths
  1. The shared inducing-point basis gives a clear way to transfer an initial function across tasks. The conditional risk bound separates approximation error, initialization error, and the effects of adaptation, providing useful insight into what additional gradient steps can and cannot improve.

  2. With the learned representation fixed, the initialization ablation shows clear benefits on all three real-world datasets, especially at small adaptation depths. The paper also reports baseline failures and limitations openly, making the results easier to assess.

Critical weaknesses
  1. The experiments do not convincingly isolate the benefit of the proposed shared initialization and finite-step adaptation. The initialization makes little difference on four synthetic task families, and a simple task-mean head performs better on Structured-Oracle. On METR-LA, validation selects T=0 in every run, so the strongest real-world result comes from the shared function without adaptation. Although the real-world initialization ablations are encouraging, comparisons with a directly trained shared head and ridge regression in the same learned space are missing. Different training budgets and independently sampled real-world test tasks also make the baseline comparisons harder to interpret.

  2. The theory provides useful insight, but its guarantees depend on assumptions that the training procedure does not establish. In particular, applying the approximation rate to the final learned representation requires Equation (23), and the proposed initialization criterion is not shown to minimize post-adaptation risk. The stopping experiments also leave its practical value unclear: the task-specific diagnostic requires 200 additional labels, while the validation-based plug-in rule offers no clear advantage over simply tuning the adaptation depth on validation tasks.

Questions for authors
  1. How does the method perform at T=0 across datasets, compared with a standard model trained on pooled data? On METR-LA, where validation selects T=0, what benefit does task adaptation provide?

  2. How much does the query-gradient term contribute at η=1e-5? An ablation with η=0 and larger values would help establish its role.

  3. How does A*_{sub} depend on p? Is there empirical evidence supporting assumption (23), comparing the final representation’s residual with that of the fixed-kernel trace minimizer?

5 comments
AuthorsOpenReview ↗

Response to Reviewer mYby (1/3)

We thank the reviewer for the detailed assessment. Below we respond to the stated weaknesses and questions. We uploaded a revised PDF in which the changes are marked in blue.


W1. The experiments do not convincingly isolate the benefit of the proposed shared initialization and finite-step adaptation. The initialization makes little difference on four synthetic task families, and a simple task-mean head performs better on Structured-Oracle. On METR-LA, validation selects T=0 in every run, so the strongest real-world result comes from the shared function without adaptation. Although the real-world initialization ablations are encouraging, comparisons with a directly trained shared head and ridge regression in the same learned space are missing. Different training budgets and independently sampled real-world test tasks also make the baseline comparisons harder to interpret.

Effect of the head initialization. We agree with the observation. This is mainly because most synthetic datasets have little structure shared across tasks, compared with the real-world datasets. In synthetic datasets, the task functions vary in amplitude and phase around a mean close to zero, so a shared initial function has little to capture. On Structured-Oracle dataset the task-mean head is better. This dataset is generated as a fixed mean function with additional task-specific variation, so a plain average of the task solutions is already close to the best shared initial function. On the real-world datasets, whose common structure is less simple, our proposed head performs better. That is, in this case the shared initial function itself is already a good predictor. Especially, in METR-LA and PEMS-BAY it is more accurate than every baseline even without any adaptation (see Q1).

Behavior on METR-LA dataset. On METR-LA the learned initial function alone is already more accurate than every other method, and validation rule selects T=0T=0. This is the case where the shared initialization is advantageous when the support set is not informative enough to improve on it, whereas the baseline models using closed-form solutions must predict solely from the support dataset.

Directly trained head and ridge regression over the learned space. A closed-form ridge regression in the learned space also requires a regularization parameter that has to be either chosen or learned, in the same way as the number of gradient steps. Moreover, in the few-shot case where support points are scarce (e.g., n=5n=5 or n=10n=10), closed-form solutions fluctuate with the choice of the regularization parameter, which carries a risk of overfitting. In our formulation, the early stopping depth TT plays a similar role of regularization, and its effect is characterized explicitly in Theorem 2 motivating the gradient-based adaptation (see also our response to Reviewer [t1RC], W1 and Q1). We agree that a comparison with a directly trained head, with the representation retrained for each choice of head, would clarify the contribution of the proposed initialization. We now state this as a limitation in Section 6 and leave it as future work.

Training budgets and test tasks. As pointed out, we have re-evaluated all methods on the same validation and test tasks on all the datasets. The ranking between the baselines remains unchanged. In addition, MAML is now reported with E=1E=1 and E=10E=10, and for each the test depth is also selected on these validation tasks over T≤30T\le30. The results are below (revised Table 1; In Appendix, Table 9 gives the full results on the depth selection of MAML).

SinusoidalOmegaMisfitOracleStructuredAir qualityMETR-LAPEMS-BAY
Ours, val$(T^\ast)$0.03550.17800.22360.42970.35920.05030.34630.0514
Ours, bound$(T^\ast)$0.04390.17920.34320.44670.36150.05270.38570.0513
MAML (E=1E=1), T=1T=11.08871.46401.25081.55130.73630.05040.38850.0904
MAML (E=1E=1), val$(T^\ast)$0.44540.69430.61480.70050.63780.04220.38850.0652
MAML (E=10E=10), T=10T=100.37760.61150.50881.20060.51400.05140.40470.0846
MAML (E=10E=10), val$(T^\ast)$0.29700.51320.44130.92540.49610.04700.39810.0847
R2-D20.04640.20250.21580.47140.62090.06310.42040.0975
iKML2.87621.41322.95320.29831.33120.07280.69640.1294
DKT0.19370.18921.20800.29590.50550.13540.81040.2639

Regarding the budgets, each baseline follows its official or standard training settings. R2-D2 and iKML use the official configuration of the iKML implementation, and MAML and DKT use the settings in Appendix F.3. All settings are listed in Table 5.

AuthorsOpenReview ↗

Response to Reviewer mYby (2/3)

W2. The theory provides useful insight, but its guarantees depend on assumptions that the training procedure does not establish. In particular, applying the approximation rate to the final learned representation requires Equation (23), and the proposed initialization criterion is not shown to minimize post-adaptation risk. The stopping experiments also leave its practical value unclear: the task-specific diagnostic requires 200 additional labels, while the validation-based plug-in rule offers no clear advantage over simply tuning the adaptation depth on validation tasks.

Regarding equation (23). Proposition 1 provides the approximation rate for the empirical trace minimizer of Stage I. Equation (23) is the condition stating how this rate can be applied to the final representation after meta-training. We consider it as a reasonable assumption for such reasons: First, meta-training starts from the representation obtained in Stage I. At this point the residual is close to the right-hand side of (23), so the condition only asks that meta-training does not worsen it. The residual is also part of the query error that adaptation cannot reduce, so minimizing the post-adaptation risk does not favor increasing it. However, we acknowledge that this is not a theoretical guarantee, and we state it as an assumption. It is used only in Corollary 5. Theorems 2 and 3 hold without it.

Initialization criterion. The first term of the criterion (8) is the adaptation energy

P^E,M(ϑ,β)=1M∑m=1M∥β−Am,Eϑ(β)∥22,\widehat{\mathcal P}_{E,M}(\vartheta,\beta)=\frac{1}{M}\sum_{m=1}^{M}\left\Vert\beta-\mathcal A_{m,E}^{\vartheta}(\beta)\right\Vert_2^2, which enters the bound of Theorem 3 through the coefficient A′=CKR02log⁡(2p)2eM+EP^E,M(ϑ^,β^0)4e(1−(1−τμ)E)2.A'=\frac{C_K R_0^2\sqrt{\log(2p)}}{2e\sqrt{M}}+\frac{\mathbb{E}\widehat{\mathcal P}_{E,M}(\widehat\vartheta,\widehat\beta_0)}{4e\big(1-(1-\tau\mu)^E\big)^2}. The criterion is chosen to reduce the second term of A′A', which is the part that depends on the initial head. The revised Section 3.2 explicitly defines this quantity and states this relationship.

Practical stopping rule. We acknowledge that the current stopping rule is more a theoretical result than a practical stopping rule. Calculating our stopping rule needs additional data, and the plug-in bound does not necessarily improve over choosing the depth with validation tasks. However, the median regret of the bound-selected depth is 1.05–1.13 on four of the five synthetic families, similar to the depth calculated using validation losses (Section 5.3, Appendix F.7). The bound is meant to show why gradient-based adaptation should stop at a finite depth and how the initial function changes the depth. A practical and accurate rule for choosing the depth for each task is left as future work.

AuthorsOpenReview ↗

Response to Reviewer mYby (3/3)

Q1. How does the method perform at T=0 across datasets, compared with a standard model trained on pooled data? On METR-LA, where validation selects T=0, what benefit does task adaptation provide?

At T=0T=0 the prediction is the shared initial function itself. Its test MSE is 4.19, 4.07, 4.23, 5.15 and 1.24 on Sinusoidal, Omega, Misfit, Oracle and Structured-Oracle, and 0.108, 0.346 and 0.052 on Air quality, METR-LA and PEMS-BAY. On the synthetic datasets a single function cannot fit tasks that vary in amplitude and phase, and adaptation is necessary. On the real collections the shared function is already a strong predictor: adaptation halves the error on Air quality, and on METR-LA and PEMS-BAY it adds little, while the shared function itself is more accurate than every baseline. A model trained on pooled data would be a useful reference for separating the effect of the shared function from that of pooling the data. Its accuracy, however, depends on the architecture and training choices, and we did not find a single setting that would make it a fair reference for all methods in the rebuttal time available. We will consider adding such a comparison later.


Q2. How much does the query-gradient term contribute at η=1e-5? An ablation with η=0 and larger values would help establish its role.

The query-gradient term comes from the first-order expansion of the one-step MAML objective from which the criterion is derived (Appendix C), and we keep it so that the criterion remains faithful to the objective. Its weight is kept small enough that the leading term (adaptation energy) remains dominant during the training and to ensure the strict convexity of the criterion. Re-computing the head of the trained models with η=0\eta=0 changes it by 0.12% on average and at most 0.35% in RKHS norm, the test MSE changes by at most 0.0002, and the selected depths are unchanged. The criterion stays strictly convex for η\eta up to about 8×10−48\times10^{-4}, and larger values degrade the error on most datasets.


Q3. How does A*_{sub} depend on p? Is there empirical evidence supporting assumption (23), comparing the final representation’s residual with that of the fixed-kernel trace minimizer?

By definition, Asub⋆=E[1M∑m=1M(L^Dm(fβ^m)−L^Dm(fSm⋆))∣FVI],β^m∈arg⁡min⁡βL^Dm(fβ).\mathcal A^\star_{\rm sub}=\mathbb E\Big[\frac1M\sum_{m=1}^M\Big(\widehat L_{D_m}\big(f_{\widehat\beta_m}\big)-\widehat L_{D_m}\big(f^\star_{S_m}\big)\Big)\Big|\mathcal F_{\rm VI}\Big],\qquad \widehat\beta_m\in\arg\min_\beta\widehat L_{D_m}(f_\beta). In our setting n<pn<p. When the support features ΦDm∈Rn×p\Phi_{D_m}\in\mathbb R^{n\times p} have full row rank, there is a β\beta with ΦDmβ=Ym\Phi_{D_m}\beta=Y_m, so L^Dm(fβ^m)=0\widehat L_{D_m}(f_{\widehat\beta_m})=0. Since L^Dm(fSm⋆)=12n∑i=1nεm,i2\widehat L_{D_m}(f^\star_{S_m})=\frac1{2n}\sum_{i=1}^n\varepsilon_{m,i}^2, Asub⋆=−E[12n∑i=1nεm,i2]∈[−σ22,0].\mathcal A^\star_{\rm sub}=-\mathbb E\Big[\frac1{2n}\sum_{i=1}^n\varepsilon_{m,i}^2\Big]\in\Big[-\frac{\sigma^2}{2},0\Big]. The term is therefore nonpositive and can be dropped from the upper bound, independently of pp.

For (23), our reasons for considering it reasonable are given in the response to W2.

Reviewer_t1RCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a kernel-based meta-learning framework for few-shot regression that jointly learns a shared function space and a transferable initial function. To enable a common initialization across tasks, the authors construct a finite-dimensional RKHS using learnable inducing points and whitened kernel features. The proposed method first employs a sparse variational objective to pre-train the feature extractor, kernel parameters, and inducing points. It then computes a shared initial prediction head through a quadratic surrogate objective involving task adaptation displacement and query-gradient information, while further optimizing the shared representation using post-adaptation query losses. At meta-test time, the learned representation is fixed, and task-specific predictions are obtained through finite-step functional gradient descent from the shared initialization. The paper also provides theoretical analyses of induced-space approximation error, conditional and uniform excess transfer risk, and early-stopping behavior.

Critical strengths
  • Rigorous integration of kernel methods and initialization-based meta-learning. The paper leverages inducing points and whitened kernel features to build a coherent framework for learning a shared function space and a transferable initialization. This elegant parameterization translates functional gradient descent into finite-dimensional linear updates, allowing the shared initial head to be solved via a closed-form quadratic optimization. This makes task adaptation analytically tractable and provides a solid foundation for further theoretical analysis.
  • Through conditional excess-risk analysis, the paper finely decomposes prediction error into representation error, initialization error in both adaptable and non-adaptable directions, and noise effects.
Critical weaknesses
  • Unclear Motivation and Insufficient Justification of the Model Design.

The paper combines kernel methods with gradient-based meta-learning, but this combination fails to fully leverage a key advantage of kernel methods: enabling task adaptation via closed-form solutions. The paper does not clearly explain why this combination is necessary, nor does it sufficiently justify why gradient-based adaptation should be preferred over closed-form methods. Furthermore, the proposed predictor can be simply expressed as (f(x)=\phi(x)^\top\beta), essentially an adaptable linear head on learned RBF feature representations. The paper needs to justify the advantages of adopting such a complex parameterization and conduct proper component ablations.

  • Complex Training Pipeline with Limited Empirical Benefits.

The proposed framework involves learnable inducing points, whitening, sparse variational pre-training, a quadratic initialization objective, and an additional meta-training procedure. Despite this complexity, the performance improvements over existing methods are limited and inconsistent. Notably, on the Oracle dataset, where the underlying functions are generated from fixed RBF kernel basis functions, the proposed method performs substantially worse than DKT and iKML (MSE: 0.4297 vs. 0.2959 and 0.2983). Furthermore, the existing ablation studies do not sufficiently isolate the contributions of individual components, making it difficult to determine whether the additional complexity is justified.

Questions for authors
  • Motivation.

Given that existing kernel-based meta-learning methods already enable closed-form task adaptation, what specific limitations motivate introducing a learned shared initialization and finite-step gradient descent? Why is this approach preferable to closed-form kernel regression with a learned prior mean?

  • Necessity of Model Design and Ablation Studies.

The proposed method essentially learns a finite-dimensional kernel feature representation with an adaptable linear head, i.e., f(x)=ϕ(x)⊤βf(x)=\phi(x)^\top\beta. Why is it necessary to simultaneously introduce learnable inducing points ZZ, a neural feature mapping hψ(x)h_\psi(x), kernel whitening, and variational pre-training? Could the authors provide more controlled ablations to isolate the individual contributions of these components? Furthermore, the shared-head ablation fixes the representation optimized for the proposed initialization. Why not retrain the representation separately for each initialization strategy to ensure a fairer comparison?

  • Justification of Equation (8).

Why is the query-gradient term derived with respect to the unwhitened variable (\alpha) rather than the actual adaptation variable β\beta? How is this difference in gradient geometry justified? Moreover, directly minimizing the post-adaptation query loss also yields a quadratic objective with a closed-form solution. Why is the proposed surrogate preferable?

  • Oracle Performance.

On the Oracle dataset, where the underlying functions are generated from a fixed RBF basis, the proposed method performs substantially worse than DKT and iKML. How do the authors explain this result? Does it reveal limitations of the learned induced space or initialization mechanism?

  • Baseline Tuning and Fairness.

Were R2-D2 and iKML tuned separately for each dataset? Given that iKML fails in 6 out of 15 synthetic runs, while its successful runs achieve lower MSE than the proposed method's reported mean on all five synthetic datasets, how do the authors ensure a fair and reliable comparison? Why does the proposed method select the adaptation depth using validation tasks, while MAML uses a fixed depth of 10 in the main comparison?

5 comments
AuthorsOpenReview ↗

Response to Reviewer t1RC (1/3)

We thank the reviewer for the careful reading and assessment. Below we provide the response to the stated weaknesses and questions. We uploaded a revised PDF in which the changes are marked in blue.


W1. Unclear Motivation and Insufficient Justification of the Model Design. The paper combines kernel methods with gradient-based meta-learning, but this combination fails to fully leverage a key advantage of kernel methods: enabling task adaptation via closed-form solutions. The paper does not clearly explain why this combination is necessary, nor does it sufficiently justify why gradient-based adaptation should be preferred over closed-form methods. Furthermore, the proposed predictor can be simply expressed as (f(x)=\phi(x)^\top\beta), essentially an adaptable linear head on learned RBF feature representations. The paper needs to justify the advantages of adopting such a complex parameterization and conduct proper component ablations.

The primary goal of our work is gradient-based adaptation in the function space. A shared initial function is learned across tasks, and a new task is fitted using few functional gradient steps. In existing kernel meta-learning methods, the predictors are often expressed as the linear combination of kernel basis functions centered at the support inputs. Hence, the head coefficients carry a different meaning in each task, making hard to define meaningful shared initial function suited for gradient based adaptation. Introducing inducing points provides bases that remain common to all tasks, while the whitening makes the Euclidean norm of the head equal to the RKHS norm, allowing us to interpret the functional gradient descent in the shared space as the standard gradient descent on the head.

We acknowledge that closed-form solutions of kernel methods are advantageous in several cases. However, they require a regularization parameter that also has to be either chosen or learned, in the same way as the number of gradient steps. Moreover, in the few-shot regime where support points are scarce (e.g., n=5n=5 or n=10n=10), closed-form solutions fluctuate with the choice of the regularization parameter, which carries a risk of overfitting. In our formulation, the early stopping depth TT plays a similar role of regularization, and its effect is characterized explicitly in Theorem 2. Indeed, for each task the initial error decreases with TT while the noise and residual terms increase with TT. Theorem 3 provides the trade-off for the risk averaged over tasks, and Corollary 4 gives the depth that balances it. For such a reason, we included R2-D2 and iKML as the representative kernel meta-learning methods that adapt by closed-form kernel ridge regression. Both are implemented along with learning the ridge parameter jointly with the representation during meta-training, and our method still attains lower error on most datasets.

We also agree that the predictor is linear in β\beta, which is intended as its linearity makes the adaptation to be interpreted as a linear recursion with a closed-form trajectory. However, the key features are the kernels at the learned inducing points, so the hypothesis space is a subspace of the RKHS of the deep kernel and its approximation error is the quantity controlled by the Stage I pre-training. The role of each component and the corresponding ablations are given in our response to W2 and Q2.


W2. Complex Training Pipeline with Limited Empirical Benefits. The proposed framework involves learnable inducing points, whitening, sparse variational pre-training, a quadratic initialization objective, and an additional meta-training procedure. Despite this complexity, the performance improvements over existing methods are limited and inconsistent. Notably, on the Oracle dataset, where the underlying functions are generated from fixed RBF kernel basis functions, the proposed method performs substantially worse than DKT and iKML (MSE: 0.4297 vs. 0.2959 and 0.2983). Furthermore, the existing ablation studies do not sufficiently isolate the contributions of individual components, making it difficult to determine whether the additional complexity is justified.

Although the training pipeline appears to have many components, each of them is required by the construction. The feature network forms the deep kernel, as in DKT, and the inducing points define the coordinate shared across tasks. The whitening is a choice of coordinate in which Euclidean gradient descent on the head matches functional gradient descent in the induced RKHS. Stage I initializes the inducing points before meta-training. Without Stage I, the error on synthetic datasets rises from 0.03-0.37 to 0.9-2.9. For the number of inducing points, the error remains lowest at p=32p=32 and increases for larger pp (Figure 6). This observation remains consistent with the tradeoff stated in Corollary 5. Oracle dataset result is discussed in Q4.

AuthorsOpenReview ↗

Response to Reviewer t1RC (2/3)

Q1. Motivation. Given that existing kernel-based meta-learning methods already enable closed-form task adaptation, what specific limitations motivate introducing a learned shared initialization and finite-step gradient descent? Why is this approach preferable to closed-form kernel regression with a learned prior mean?

Please also refer to our response to W1. A closed-form solution with a learned prior mean even requires a ridge parameter, either chosen or learned without a characterization of how it affects the error of each task. In gradient-based adaptation the corresponding quantity is the depth TT, and we characterize its effect via risk bound analysis. Indeed, for each task, the initial error decreases with TT while the noise and residual terms increase with TT. Theorem 3 provides the similar trade-off for the risk average over tasks, and Corollary 4 gives the depth that balances them. To our knowledge, prior kernel meta-learning work does not provide such an analysis, and it is where gradient-based adaptation makes it possible.


Q2. Necessity of Model Design and Ablation Studies. The proposed method essentially learns a finite-dimensional kernel feature representation with an adaptable linear head, i.e., f(x)=ϕ(x)⊤βf(x)=\phi(x)^\top \beta Why is it necessary to simultaneously introduce learnable inducing points ZZ , a neural feature mapping hψ(x)h_\psi(x) , kernel whitening, and variational pre-training? Could the authors provide more controlled ablations to isolate the individual contributions of these components? Furthermore, the shared-head ablation fixes the representation optimized for the proposed initialization. Why not retrain the representation separately for each initialization strategy to ensure a fairer comparison?

We discuss the component ablations in our response to W2. Considering a neural feature mapping makes the model more flexible, and variational pre-training is necessary to determine the best inducing points as well known in the study of Gaussian Process Regression. The model flexibility and learned inducing points can make the induced space better suited for the hypothesis space of given tasks. For the shared-head ablation, we meta-train the representation with our proposed head and replace only the heads at test time. This choice was intended to analyze how the choice of initialization matters within the same learned induced space. This design matches the decomposition in Theorem 3: under the fixed induced space, the approximation error remains constant, while our criterion for the head decreases the adaptation-energy term appearing in the risk bound. We agree that a comparison of the full models with retrained representation for each head, will provide the initialization along with the space optimized for it. We clarified the scope of the current ablation and leave this comparison for future work.


Q3. Justification of Equation (8). Why is the query-gradient term derived with respect to the unwhitened variable (\alpha) rather than the actual adaptation variable β\beta? How is this difference in gradient geometry justified? Moreover, directly minimizing the post-adaptation query loss also yields a quadratic objective with a closed-form solution. Why is the proposed surrogate preferable?

The query-gradient correction in Eq. (8) is motivated by a first-order expansion of the one-step MAML objective in the inducing coefficients α\alpha. Task adaptation uses the whitened coefficients β\beta. By the chain rule, ∇αL=KZZ1/2∇βL\nabla_\alpha\mathcal L=K_{ZZ}^{1/2}\nabla_\beta\mathcal L, so the correction measures ∇βL⊤KZZ∇βL\nabla_\beta\mathcal L^\top K_{ZZ}\nabla_\beta\mathcal L, the squared functional gradient evaluated at the inducing points. Setting η=0\eta=0 would remove this correction. We use a small positive weight, η=10−5\eta=10^{-5}, to retain it while keeping the adaptation energy as the leading term and preserving strict convexity. Direct minimization of the post-adaptation query loss also admits a closed-form head. We chose Eq. (8) because its leading term is the adaptation energy, the quantity through which the initialization enters the bound of Theorem 3. We leave a comparison with the head obtained by direct minimization for future work.

AuthorsOpenReview ↗

Response to Reviewer t1RC (3/3)

Q4. Oracle Performance. On the Oracle dataset, where the underlying functions are generated from a fixed RBF basis, the proposed method performs substantially worse than DKT and iKML. How do the authors explain this result? Does it reveal limitations of the learned induced space or initialization mechanism?

Oracle dataset is a challenging setting for our method. On the same test tasks, our method achieves an MSE of 0.381 with the depth selected up to T=200T=200 (Table 6; 0.430 with the cap of T≤30T\leq 30 in Table 1), compared with 0.296 for DKT and 0.298 for iKML. The tasks are generated from a fixed RBF basis, which DKT fits exactly with an RBF Gaussian process, whereas our method adapts within a space spanned by a finite set of learned inducing points. We included this family to measure the gap to such a well-specified kernel method. We also state this purpose in Section 5.1.


Q5. Baseline Tuning and Fairness. Were R2-D2 and iKML tuned separately for each dataset? Given that iKML fails in 6 out of 15 synthetic runs, while its successful runs achieve lower MSE than the proposed method's reported mean on all five synthetic datasets, how do the authors ensure a fair and reliable comparison? Why does the proposed method select the adaptation depth using validation tasks, while MAML uses a fixed depth of 10 in the main comparison?

We address the four points in turn.

Hyperparameters of R2-D2 and iKML. Both use the official implementation of iKML with the hyperparameters of its Beijing air quality configuration, applied unchanged to all datasets (Appendix F.3). Their ridge parameter is learned jointly with the representation by the meta-training loss.

Same test tasks and the depth of MAML. In the submission the synthetic dataset comparison was already paired, but on the real-world datasets, the baselines were evaluated on test tasks drawn independently. We have re-evaluated all methods on the same validation and test tasks. In addition, MAML is now reported with E=1E=1 and E=10E=10, and for each the test depth is also selected on these validation tasks over T≤30T\le30. The results are below (revised Table 1; In Appendix, Table 9 gives the full results on the depth selection of MAML).

SinusoidalOmegaMisfitOracleStructuredAir qualityMETR-LAPEMS-BAY
Ours, val$(T^\ast)$0.03550.17800.22360.42970.35920.05030.34630.0514
Ours, bound$(T^\ast)$0.04390.17920.34320.44670.36150.05270.38570.0513
MAML (E=1E=1), T=1T=11.08871.46401.25081.55130.73630.05040.38850.0904
MAML (E=1E=1), val$(T^\ast)$0.44540.69430.61480.70050.63780.04220.38850.0652
MAML (E=10E=10), T=10T=100.37760.61150.50881.20060.51400.05140.40470.0846
MAML (E=10E=10), val$(T^\ast)$0.29700.51320.44130.92540.49610.04700.39810.0847
R2-D20.04640.20250.21580.47140.62090.06310.42040.0975
iKML2.87621.41322.95320.29831.33120.07280.69640.1294
DKT0.19370.18921.20800.29590.50550.13540.81040.2639

Selecting the depth lowers the error of MAML on most datasets, but the ranking between the baselines remains unchanged: our method attains the lowest error on five of the eight datasets, as in the submission.

Regarding iKML performance. All methods are evaluated on the same test tasks, and each mean in Table 1 includes every run. With the official training settings, 6 of the 15 iKML runs on the synthetic datasets remained near a constant value throughout training. Whether a run converged depended on the random seed, while the learned ridge parameter remained near its initial value. Excluding these runs, iKML attains a lower mean MSE than our method on the synthetic datasets. Table 4 already reports the individual runs, and Appendix F.3 attributes the failures to the initialization of the frequency network for 1-dimensional inputs. The real-world rows of Table 4 are updated to the runs evaluated on the same test tasks.

Comments on the paper

1 comment

The authors have posted their responses to the reviews. Could I please encourage the reviewers to read these responses and reply to the authors.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.