- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Geometry-Aware Teaching (GAT), which combines curvature-aware example selection with preconditioned updates in RKHS. The authors analyze the spectral limitations of existing first-order teaching methods and show that GAT can mitigate the dependence on kernel conditioning. They also provide convergence guarantees and establish finite-horizon optimality under specific assumptions. Experiments on function fitting and image reconstruction show faster convergence than first-order teaching baselines.
Critical strengths
S1. The theoretical analysis is generally well structured. The paper provides convergence guarantees with detailed proofs, and the overall theoretical development seems reasonable to me.
S2. The empirical results support the proposed method. The experiments show consistent improvements over the first-order baselines across several function fitting and reconstruction tasks.
Critical weaknesses
W1. The teaching-specific theoretical contribution is unclear.
Much of the analysis, including the spectral-bias argument and the benefits of preconditioning, seems closely related to standard kernel optimization theory. While the paper establishes convergence guarantees, I am not sure what new theoretical insight comes specifically from the teaching formulation rather than the use of a second-order learner. It would be helpful to clarify this distinction, particularly in relation to existing kernel/NTK-based analyses.
W2. The broader significance is unclear to me.
I found it difficult to see what broader insight this paper provides beyond the specific setting studied. Prior work has introduced the nonparametric teaching framework or connected it to practical neural learners. Here, the main analysis focuses on fixed-feature least squares, and the experiments are largely limited to function fitting and reconstruction. Beyond extending the framework to second-order updates, I would appreciate a clearer explanation of what new understanding of teaching or learning emerges from this work.
W3. The empirical contribution of the teaching strategy could be clearer.
The main experiments compare GAT with first-order baselines, making it difficult to separate the effects of selection and preconditioning. While Appendix F provides controlled comparisons and shows additional gains from GAT selection, these gains are relatively modest compared to those from preconditioning. I think the paper should place more emphasis on these controlled results to clarify the contribution of the teaching strategy itself.
Questions for authors
I would appreciate clarification on the points raised in the weaknesses above, particularly regarding the teaching-specific contribution and broader significance of the work.
2 comments
Response to the Review [2/2]
W3. The empirical contribution of the teaching strategy could be clearer. “these gains are relatively modest compared to those from preconditioning.”
The fixed-learner improvements are substantial and consistent, while separate controlled experiments test the theory’s predictions.
In Table 8, all selectors share the preconditioner, batch size, and step-size multiplier. At 192 presentations per photograph, GAT reduces chroma MSE relative to Greedy by 17.1% per photograph on average and improves all 200 photographs. At 384 presentations, the mean initial-error-normalized MSE decreases from 0.1058 with Greedy to 0.0633 with GAT, a 40.2% reduction. These gains come from changing the teaching strategy while keeping the learner fixed.
We additionally compare GAT, Greedy, and Random using the same preconditioned learner within each task to isolate the contribution of example selection.
On the original curve task from Figures 1–2, the pool contains 100 points and the learner uses 20 sine modes. All methods share the preconditioner, a batch size of 10, and a step-size multiplier of 1. The table reports the number of example presentations needed to reach each target reduction in initial MSE. Random results average ten runs. (Tables 10 in the revised manuscript)
| Target MSE reduction | GAT presentations | Greedy presentations | Random presentations |
|---|---|---|---|
| 90% | 50 | 60 | 127 |
| 95% | 70 | 80 | 166 |
| 99% | 110 | 120 | 249 |
GAT reaches each target with 8.3–16.7% fewer presentations than Greedy and 55.8–60.6% fewer than Random.
We also reconstruct the original 64 × 64 Cheetah image. All methods share the preconditioner, a batch size of 50, and a step-size multiplier of 1. Random PSNR averages three runs. (Tables 11 in the revised manuscript)
| Pixel presentations | GAT PSNR (dB) | Greedy PSNR (dB) | Random PSNR (dB) |
|---|---|---|---|
| 4,100 | 17.14 | 13.08 | 9.95 |
| 8,200 | 20.59 | 18.10 | 17.10 |
GAT improves PSNR over Greedy by 4.06 dB and 2.49 dB at the two budgets, corresponding to approximately 60.7% and 43.6% lower MSE. These substantial improvements arise from changing the teaching strategy while keeping the learner fixed.
The mechanism experiments explain why the selection rule matters. Figure 7 crosses selection rules with first-order and preconditioned updates and shows that their relative ordering reverses when the update geometry changes. Appendix F.2 tests Corollary 9 across kernel condition numbers from 1 to 1,000. The selected batches remain identical and predictions agree to numerical precision. These results corroborate the predicted relationship between teaching and update geometry. Together with the fixed-learner gains, they support both GAT’s joint design and the effectiveness of its teaching decisions.
Response to the Review [1/2]
We thank the reviewer for finding the theoretical analysis well structured and the overall development reasonable, with detailed proofs supporting the convergence guarantees. We also appreciate the recognition of consistent empirical improvements over baselines across function-fitting and reconstruction tasks.
The new teaching-specific insight is that update geometry determines how batch selection contributes to convergence and can make sequential example selection optimal over every finite teaching budget.
Classical NTK theory [1] characterizes how gradient descent changes network predictions. EigenPro [2] modifies kernel geometry to accelerate the learner’s updates. GAT studies how a teacher should select examples in that geometry and what the resulting adaptive decisions guarantee.
Lemmas 6–7 and Theorem 8 make this distinction explicit. Lemma 6 establishes a risk-decrease lower bound with a common coefficient across candidate batches. Exact GAT selection maximizes this guarantee. The refined convergence bound then retains the gain achieved by the actual selected batch:
&E_{t+1} \leq \left(1-\frac{\nu(2-\nu)\gamma_t}{R^2}\right)E_t,\\ &\gamma_t=\frac{q_t(\mathcal B_t^{\mathrm{GAT}})}{2E_t}\geq1, \end{aligned}$$ where, $E_t$ is the pool prediction risk, $R$ bounds the whitened batch covariance, and $\nu$ is the step-size multiplier. The teacher’s choice directly enters the contraction guarantee through $\gamma_t$. Corollary 9 establishes spectral-rescaling invariance of both the selected batches and the prediction trajectory under its fixed protocol. Proposition 10 gives a stronger teaching comparison: with the preconditioned learner and step size fixed in a complete pool space, GAT minimizes terminal risk over **all legal teaching sequences at every finite budget**. [1] Neural Tangent Kernel: Convergence and Generalization in Neural Networks. NeurIPS, 2018. [2] Diving into the Shallows: A Computational Perspective on Large-Scale Shallow Learning. NeurIPS, 2017. > W2. The broader significance is unclear to me. **The broader contribution is a principle for teaching: examples should be evaluated through the prediction corrections that the learner can obtain from them, so teaching and optimization should be designed together.** The spectral analysis identifies why selecting large functional gradients can leave important prediction errors weakly corrected. Example selection changes the batch covariance, but first-order updates still pass through the kernel scaling. GAT addresses both decisions together by evaluating batches in the geometry used for their updates. Lemma 6 connects this evaluation to a common prediction-risk decrease guarantee. Table 6 demonstrates the practical consequence: the relative ranking of Greedy and GAT selection reverses when the update geometry changes. Thus, improving the learner can also change which teaching rule is effective. The teacher and optimizer cannot be evaluated as independent sources of improvement. The analysis further explains when teaching requires joint batch decisions and when a simple sequential rule is sufficient. In general pools, the curvature-aware score contains interactions between examples, so independently ranking examples need not maximize the batch score. In a complete pool space, preconditioning makes their effects independent. Proposition 10 then proves that selecting the largest remaining residuals minimizes terminal risk over all legal teaching sequences at every finite budget. The same teacher is therefore optimal without advance knowledge of the stopping budget. These results provide concrete guidance for teaching design. The selection criterion should reflect the learner’s prediction response, and batch construction should account for interactions between examples. When the geometry removes those interactions, the analysis identifies a simple rule with an exact budget-allocation guarantee. For neural networks, future work can extend this principle by adapting the teaching criterion to the optimizer and the evolving prediction response during training.