- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies few-shot regression, where a model must adapt to a new task using only a small number of labeled examples. The proposed method learns a shared set of kernel basis functions using a neural feature extractor and inducing points. Different tasks share the same basis functions and use task-specific coefficients. The method first pre-trains the shared representation using a variational objective. During meta-training, it solves a quadratic optimization problem to compute a shared initialization for the coefficients. Each task then adapts from this initialization using gradient descent. The shared representation is further updated based on the query losses. For a new task, the model keeps the shared representation fixed and only updates the task-specific coefficients using a few labeled examples. The paper also analyzes how the number of adaptation steps affects prediction error. It derives error bounds that motivate early stopping rules. Experiments on synthetic and real-world datasets compare the method with MAML, R2-D2, DKT, and iKML. The authors also examine the effects of pre-training and shared initialization through ablation studies.
Critical strengths
-
The paper provides a structured analysis of functional gradient adaptation within the learned kernel space. The conditional and uniform excess transfer risk bounds help characterize the effects of approximation error, initialization, and adaptation depth. This analysis is a useful aspect of the work.
-
The experiments cover both synthetic and real-world regression tasks. The ablations in Section 5.4 and Appendix F provide evidence that variational pre-training and the proposed shared initialization can improve predictive performance, particularly on the real-world datasets.
Critical weaknesses
-
The paper addresses a well-studied few-shot regression setting. Its framework builds on established ideas in kernel learning and gradient-based meta-learning. The main methodological difference appears to be the quadratic objective for shared initialization. However, the paper does not sufficiently explain the advantages of this design over existing approaches.
-
The evaluation mainly compares against MAML, R2-D2, DKT, and iKML, introduced between 2017 and 2022. These are relevant baselines. However, the paper does not establish whether they are sufficient to assess the proposed method's competitiveness. A broader comparison with more recent or closely related approaches would strengthen the evaluation.
Questions for authors
-
What is the main advantage of the proposed quadratic shared initialization over existing meta-learning approaches? In particular, how does it compare with directly meta-learning the initial coefficients under the same kernel representation?
-
Table 1 uses independently sampled real-world test tasks for the proposed method and the baselines. Could the authors clarify why the same test tasks were not used for all methods?
3 comments
Thank you for the response and additional experiments. The clarifications address some of my questions, and I appreciate the re-evaluation on the same validation and test tasks and the strengthened MAML comparison. However, the advantage of the proposed quadratic initialization over directly meta-learning the initial coefficients under the same kernel representation remains unclear. My main concern about the methodological contribution therefore remains unresolved, and I maintain my original rating.
Thank you for the follow-up comment and for acknowledging the re-evaluation and the baseline comparison. We understand the remaining concern and will address it in future work, as stated in Section 6. Thank you again for your time and consideration.
We thank the reviewer for the careful reading and assessment. Below we provide the response to the stated weaknesses and questions. We uploaded a revised PDF in which the changes are marked in blue.
The quadratic objective is one part of a construction for gradient-based adaptation in kernel meta-learning. In existing kernel meta-learning methods, the coefficients are tied to the support inputs. This changes their meaning across tasks so that shared initial function is not well defined. Inducing points provide bases shared by all tasks, and in these coordinates gradient descent on the head is functional gradient descent. Within this space, the quadratic objective gives the initial head in closed form. Its first term, the adaptation energy, measures how far the initial function is located from the functions adapted with few support steps on each task. The quadratic objective reduces it, thereby lowering the risk bound (in Theorem 3).
The baselines were chosen by how closely they relate to our setting: gradient-based adaptation and kernel-based meta-learning for few-shot regression. MAML is the standard gradient-based method, and DKT, R2-D2 and iKML are representative kernel-based meta-learning methods for few-shot regression with available implementations. To our knowledge, more recent kernel-based methods for this setting are few. Recent models for few-shot regression, such as ScaML-GP (AISTATS 2024), which adapts by Gaussian-process posterior inference, and TNP (ICML 2022) with its variants, which predicts from the support set with a learned transformer encoder, are not direct counterparts of our method. Instead, we backed our comparison with the MAML. The revised Table 1 now reports MAML trained with one and ten inner steps, each with its test depth also selected by minimizing the validation loss.
For a given representation, the head is available in closed form. Its leading term is the adaptation energy, where reducing it lowers the risk bound in Theorem 3. Thus, such choice of the head is tied to the analysis. We do not claim an empirical advantage over a head trained directly. A comparison with such a head, with the representation retrained for each choice is left as future work. We also state it as a limitation in Section 6.
We agree that all methods should be evaluated on the same test tasks. We have re-evaluated all methods on the same validation and test tasks on the real-world datasets. In addition, MAML is now reported with E=1 and E=10, and for each the test depth is also selected on these validation tasks over T≤30. The results are below (revised Table 1; In Appendix, Table 9 gives the full results on the depth selection of MAML). The ranking between baselines is unchanged.