- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates in-context learning on single-index nonlinear tasks. It compares two architectures based on a single layer of linear attention: kernel learners, which apply a fixed nonlinear feature mapping before attention, and feature learners, which apply a learnable nonlinear readout after attention. The authors use the replica method to predict memorization and generalization errors and validate these predictions through synthetic experiments. The paper also rigorously characterizes the distinct scaling laws governing the two learners’ context requirements in the thermodynamic limit.
Critical strengths
-
Mathematically interpretable construction. Using a simplified model with a single layer of linear attention, the paper analyzes how pretraining scale, task diversity, and context length jointly affect the performance of both learners. Distinguishing memorization error on previously seen tasks from generalization error on novel tasks reveals the respective strengths of the two architectures in task memorization and generalization across tasks.
-
Validation of theoretical predictions at multiple levels. The experiments cover pretraining sizes below and above each learner’s interpolation threshold, testing whether the replica predictions capture double descent and changes in relative performance. The appendix further compares theoretical order parameters with their empirical counterparts and examines different task pool sizes. These comparisons support the analysis through both prediction errors and internal statistical quantities.
Critical weaknesses
-
Limited experimental scope and assessment of statistical stability. The numerical experiments primarily use a fixed teacher function comprising first- and second-order components, without systematically evaluating other nonlinear teacher functions that satisfy the theoretical assumptions. The accuracy of the predictions across different nonlinear structures therefore remains unclear. Moreover, Figures 2 and 5 report results from single training runs, without error bars or confidence intervals across repeated runs. This limits the assessment of statistical stability, particularly near the interpolation peaks and in regimes where the performance gap between the methods is small.
-
Missing direct experimental validation of predictions under context length mismatch. Section 4.2 uses the replica phase diagrams in Figures 3(c,d) to discuss how differences between training and inference context lengths affect the relative advantages of the two learners. However, the experiments in Figure 4 are restricted to equal training and inference lengths (), leaving the predictions concerning context length mismatch without direct experimental validation.
Questions for authors
Have you conducted numerical experiments with mismatched training and inference context lengths ()? If so, do the empirical transitions between regimes favoring the kernel learner and the feature learner agree with the boundaries predicted in Figures 3(c,d)?
2 comments
Thanks for the response. The response clarified my questions.
(W1) Limited experimental scope and assessment of statistical stability. The numerical experiments primarily use a fixed teacher function comprising first- and second-order components, without systematically evaluating other nonlinear teacher functions that satisfy the theoretical assumptions. The accuracy of the predictions across different nonlinear structures therefore remains unclear. Moreover, Figures 2 and 5 report results from single training runs, without error bars or confidence intervals across repeated runs. This limits the assessment of statistical stability, particularly near the interpolation peaks and in regimes where the performance gap between the methods is small.
Reply to W1: To address this question, we consider a normalized sinusoidal teacher in Appendix C.8, Figure 13, which has nonzero third- and higher-order Hermite coefficients. The numerics match the replica predictions, and the performance of these learners remains qualitatively the same.
Furthermore, we have updated Figures 2, 5, and 6 to report means and standard deviations across three independent training runs, which shows good agreement with our theory.
(W2) Missing direct experimental validation of predictions under context length mismatch. Section 4.2 uses the replica phase diagrams in Figures 3(c,d) to discuss how differences between training and inference context lengths affect the relative advantages of the two learners. However, the experiments in Figure 4 are restricted to equal training and inference lengths (L=L~), leaving the predictions concerning context length mismatch without direct experimental validation.
(Q1) Have you conducted numerical experiments with mismatched training and inference context lengths (L=L~)? If so, do the empirical transitions between regimes favoring the kernel learner and the feature learner agree with the boundaries predicted in Figures 3(c,d)?
Reply to W2 and Q1: We added empirical phase diagrams for memorization and generalization on an 8×8 grid of training and inference context lengths at d=32 and d=24 (Appendix C.7, Figures 10 and 11). The measured diagrams at both dimensions agree well with the replica predictions in Figure 3.
Moreover, in the revision we have also substantially strengthened Theorem 1 to rigorously characterize the population errors for arbitrary training and inference context lengths L and L, which provides a rigorous explanation for the context-length mismatch mechanisms underlying Figures 3(c,d), independently of the replica method and Gaussian-equivalence assumptions.