acceptodds
Under review as a conference paper at ICLR 2027

In-context Learning of Single-index Targets: Comparing Kernel and Feature Learners

Abstract

In-context learning (ICL) enables a pretrained model to infer a task from demonstrations without updating its parameters. While much of the existing theory focuses on linear target functions, in this paper we study nonlinear cases by comparing two one-layer attention architectures on the same family of single-index tasks. A kernel learner first maps inputs through a fixed nonlinear feature map and then applies linear attention, whereas a feature learner applies attention to the original input, followed by a learned nonlinear readout. We derive predictions for their memorization and generalization errors using the replica method, retaining the effects of pretraining size, task-pool diversity, and training and inference context lengths. The resulting predictions closely match numerical experiments across a broad range of regimes. Our analysis yields phase diagrams that characterize when each architecture is advantageous as the amount of pretraining data, task diversity, and context lengths vary. We further identify qualitatively different context-length scalings for the two learners. Together, these results clarify how architectural choices interact with the dataset and govern nonlinear in-context learning.

Primary area:
learning theory
Keywords:
in context learning, replica method, kernel learning, feature learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HZanOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates in-context learning on single-index nonlinear tasks. It compares two architectures based on a single layer of linear attention: kernel learners, which apply a fixed nonlinear feature mapping before attention, and feature learners, which apply a learnable nonlinear readout after attention. The authors use the replica method to predict memorization and generalization errors and validate these predictions through synthetic experiments. The paper also rigorously characterizes the distinct scaling laws governing the two learners’ context requirements in the thermodynamic limit.

Critical strengths
  1. Mathematically interpretable construction. Using a simplified model with a single layer of linear attention, the paper analyzes how pretraining scale, task diversity, and context length jointly affect the performance of both learners. Distinguishing memorization error on previously seen tasks from generalization error on novel tasks reveals the respective strengths of the two architectures in task memorization and generalization across tasks.

  2. Validation of theoretical predictions at multiple levels. The experiments cover pretraining sizes below and above each learner’s interpolation threshold, testing whether the replica predictions capture double descent and changes in relative performance. The appendix further compares theoretical order parameters with their empirical counterparts and examines different task pool sizes. These comparisons support the analysis through both prediction errors and internal statistical quantities.

Critical weaknesses
  1. Limited experimental scope and assessment of statistical stability. The numerical experiments primarily use a fixed teacher function comprising first- and second-order components, without systematically evaluating other nonlinear teacher functions that satisfy the theoretical assumptions. The accuracy of the predictions across different nonlinear structures therefore remains unclear. Moreover, Figures 2 and 5 report results from single training runs, without error bars or confidence intervals across repeated runs. This limits the assessment of statistical stability, particularly near the interpolation peaks and in regimes where the performance gap between the methods is small.

  2. Missing direct experimental validation of predictions under context length mismatch. Section 4.2 uses the replica phase diagrams in Figures 3(c,d) to discuss how differences between training and inference context lengths affect the relative advantages of the two learners. However, the experiments in Figure 4 are restricted to equal training and inference lengths (L=L~L=\tilde{L}), leaving the predictions concerning context length mismatch without direct experimental validation.

Questions for authors

Have you conducted numerical experiments with mismatched training and inference context lengths (L≠L~L\neq\tilde{L})? If so, do the empirical transitions between regimes favoring the kernel learner and the feature learner agree with the boundaries predicted in Figures 3(c,d)?

2 comments

(W1) Limited experimental scope and assessment of statistical stability. The numerical experiments primarily use a fixed teacher function comprising first- and second-order components, without systematically evaluating other nonlinear teacher functions that satisfy the theoretical assumptions. The accuracy of the predictions across different nonlinear structures therefore remains unclear. Moreover, Figures 2 and 5 report results from single training runs, without error bars or confidence intervals across repeated runs. This limits the assessment of statistical stability, particularly near the interpolation peaks and in regimes where the performance gap between the methods is small.

Reply to W1: To address this question, we consider a normalized sinusoidal teacher in Appendix C.8, Figure 13, which has nonzero third- and higher-order Hermite coefficients. The numerics match the replica predictions, and the performance of these learners remains qualitatively the same.

Furthermore, we have updated Figures 2, 5, and 6 to report means and standard deviations across three independent training runs, which shows good agreement with our theory.

(W2) Missing direct experimental validation of predictions under context length mismatch. Section 4.2 uses the replica phase diagrams in Figures 3(c,d) to discuss how differences between training and inference context lengths affect the relative advantages of the two learners. However, the experiments in Figure 4 are restricted to equal training and inference lengths (L≠L~L\neq\tilde{L}), leaving the predictions concerning context length mismatch without direct experimental validation.

(Q1) Have you conducted numerical experiments with mismatched training and inference context lengths (L≠L~L\neq\tilde{L})? If so, do the empirical transitions between regimes favoring the kernel learner and the feature learner agree with the boundaries predicted in Figures 3(c,d)?

Reply to W2 and Q1: We added empirical phase diagrams for memorization and generalization on an 8×88\times8 grid of training and inference context lengths at d=32d=32 and d=24d=24 (Appendix C.7, Figures 10 and 11). The measured diagrams at both dimensions agree well with the replica predictions in Figure 3.

Moreover, in the revision we have also substantially strengthened Theorem 1 to rigorously characterize the population errors for arbitrary training and inference context lengths LL and L~\widetilde L, which provides a rigorous explanation for the context-length mismatch mechanisms underlying Figures 3(c,d), independently of the replica method and Gaussian-equivalence assumptions.

Thanks for the response. The response clarified my questions.

Reviewer_ns1fOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates nonlinear in-context learning (ICL) on single-index regression tasks by comparing two one-layer attention architectures: a kernel learner, which applies a fixed nonlinear feature mapping before attention, and a feature learner, which applies a learned nonlinear readout after attention. It examines how pretraining sample size, task diversity, and training/inference context lengths affect memorization of previously seen tasks and generalization to unseen tasks.

Critical strengths

Novel theoretical insights into nonlinear ICL and context-length scaling. The paper establishes a meaningful theoretical distinction between kernel-based and feature-learning architectures. In particular, the rigorous result showing that kernel learners require L=Θ(dk)L=\Theta(d^k) to recover degree-kk components, whereas feature learners can exploit all nonlinear components at L=Θ(d)L=\Theta(d), provides a valuable understanding of architectural inductive biases and sample efficiency. This theoretical contribution is the strongest reason supporting acceptance.

Critical weaknesses

Limited generality and practical relevance of the theoretical findings. The analysis is restricted to simplified one-layer linear attention architectures, spherical-harmonic feature mappings, and synthetic single-index regression tasks. Moreover, the feature learner employs an idealized two-stage training procedure with a population-optimal nonlinear readout. These assumptions substantially differ from practical Transformer architectures and end-to-end training. Consequently, it remains unclear whether the identified architectural advantages and context-length scaling behaviors extend to realistic ICL settings, limiting the broader significance of the findings.

Reliance on non-rigorous assumptions for the main performance characterization. Although the context-length scaling theorem is rigorously established under specific asymptotic conditions, the main memorization and generalization error predictions rely on the replica method and an unproven Gaussian-equivalence assumption. The numerical validation supports these approximations but covers a relatively narrow range of synthetic settings. This limits confidence in the general validity of the predicted phase diagrams and performance trade-offs, particularly outside the examined parameter regimes.

Questions for authors
  1. The main memorization and generalization predictions rely on the replica method and a Gaussian-equivalence assumption. Can the authors clarify the conditions under which these approximations are expected to hold, particularly near interpolation thresholds or under limited task diversity? Additional justification or evidence regarding their reliability would materially strengthen confidence in the theoretical conclusions.
  2. The feature learner uses an idealized two-stage training procedure with a population-optimal nonlinear readout, while the two architectures also differ substantially in parameter count. To what extent do the reported performance advantages arise from architectural differences rather than training procedures or model capacity? Clarifying the sensitivity to finite readout-training samples and matched computational or parameter budgets would help establish the robustness of the central conclusions.
6 comments

(W1) Limited generality and practical relevance of the theoretical findings. The analysis is restricted to simplified one-layer linear attention architectures, spherical-harmonic feature mappings, and synthetic single-index regression tasks. Moreover, the feature learner employs an idealized two-stage training procedure with a population-optimal nonlinear readout. These assumptions substantially differ from practical Transformer architectures and end-to-end training. Consequently, it remains unclear whether the identified architectural advantages and context-length scaling behaviors extend to realistic ICL settings, limiting the broader significance of the findings.

Reply to W1: We agree that our analysis focuses on simplified attention architectures. However, such controlled settings are standard in the theoretical study of ICL. For example, Lu et al. (2025) and Takanami et al. (2026) analyze one-layer linear attention on linear regression tasks. Our work extends this framework to nonlinear single-index regression, revealing a qualitatively new architectural distinction: kernel learners recover different harmonic components at distinct context-length scales, whereas feature learners can exploit all Hermite components at the same scale. This separation is rigorously established for a broad class of nonlinear teacher functions.

To investigate the robustness of our conclusions beyond the idealized setting, we have added several experiments. First, end-to-end training of the attention matrix and nonlinear readout yields similar errors to the two-stage procedure (Figure 6) and largely preserves the predicted phase diagrams (Figure 12). Second, experiments with a sinusoidal teacher containing higher-order Hermite components remain consistent with our replica predictions (Figure 13). These results suggest that our findings are not specific to the two-stage training procedure or the quadratic teacher.

While extending these conclusions to standard Transformers remains an open question, our simplified setting provides a tractable framework for understanding how the placement of nonlinearity relative to attention affects nonlinear ICL.

(W2) Reliance on non-rigorous assumptions for the main performance characterization. Although the context-length scaling theorem is rigorously established under specific asymptotic conditions, the main memorization and generalization error predictions rely on the replica method and an unproven Gaussian-equivalence assumption. The numerical validation supports these approximations but covers a relatively narrow range of synthetic settings. This limits confidence in the general validity of the predicted phase diagrams and performance trade-offs, particularly outside the examined parameter regimes.

Reply to W2: We agree that the finite-pretraining predictions rely on non-rigorous approximations, and we would like to clarify several reasons supporting their validity.

First, the optimization problems considered in our main experiments are convex. For the feature learner, we take the Stage-I readout to be σ0(u)=u\sigma_0(u)=u, so with positive regularization, the objectives are strictly convex and have unique global minimizers. This makes the replica-symmetric ansatz particularly natural.

Second, our replica predictions admit an independent rigorous consistency check. The formulas obtained from the replica equations in the infinite-pretraining and infinite-task-diversity limit coincide with Theorem 1, which is proved directly from the original optimization problems in Appendices D–E, without relying on replica or Gaussian-equivalence assumptions. This agreement provides nontrivial evidence for the predictions.

Third, recent theoretical developments provide further motivation for Gaussian universality in structured high-dimensional models. For example, Nguyen, Lu, and Sen (2026, arXiv:2610.05327) establish universality for AMP with rank-one quadratic sensing matrices, despite strong dependence among their coordinates, which is linked to the universality in ERM (e.g. arxiv:2505.17958).

Finally, we have further strengthened our rigorous analysis by extending Theorem 1 to arbitrary training and inference context lengths LL and L~\widetilde L. These results agree with the corresponding predictions of our replica analysis, providing a stronger independent consistency check.

We believe that a rigorous treatment of finite-sample performance is possible in principle. When σ0(u)=u\sigma_0(u)=u, the ridge regression has exact solutions. One could therefore attempt to establish the asymptotic predictions using random matrix theory. The main difficulty is controlling the non-Gaussian features and the correlations induced by the task pool. This is a substantial technical problem that we leave for future work.

We broadened the numerical validation in several ways. In Appendix C.7, we compare the replica phase diagrams (Figure 3) with experiments at d=24 and d=32, which shows good agreement. In Appendix C.8, we consider a sinusoidal teacher containing higher-order Hermite components, and we find that the numerics are still consistent with the replica prediction. These results support the validity of the replica predictions across a broader range of settings.

(Q1) The main memorization and generalization predictions rely on the replica method and a Gaussian-equivalence assumption. Can the authors clarify the conditions under which these approximations are expected to hold, particularly near interpolation thresholds or under limited task diversity? Additional justification or evidence regarding their reliability would materially strengthen confidence in the theoretical conclusions.

Reply to Q1: As discussed in our response to W2, the convexity of the optimization problems, the agreement with our independent rigorous analysis, and recent developments in Gaussian universality provide several reasons to support our replica predictions.

We expect these predictions to remain asymptotically valid in the high-dimensional regime with positive regularization, including near interpolation thresholds. As shown in Figure 2 and Appendix C.1, the numerical predictions agree well with experiments overall, with some visible deviations near the interpolation peaks. We believe these discrepancies may arise from finite-dimensional corrections and possible numerical optimization errors. These deviations therefore do not necessarily indicate a breakdown of the asymptotic predictions.

Regarding limited task diversity, our analysis explicitly incorporates the task pool through its empirical covariance structure. Figure 2 and Appendix C.2 already show good agreement with experiments for small task pools (currently only 2424 tasks). We expect the predictions to remain applicable for fixed positive task-diversity ratios, although the finite-size effects might be stronger for smaller task-diversity.

(Q2) The feature learner uses an idealized two-stage training procedure with a population-optimal nonlinear readout, while the two architectures also differ substantially in parameter count. To what extent do the reported performance advantages arise from architectural differences rather than training procedures or model capacity? Clarifying the sensitivity to finite readout-training samples and matched computational or parameter budgets would help establish the robustness of the central conclusions.

Reply to Q2: We thank the reviewer for raising this important question. We would like to distinguish the effects of the training procedure from those of model capacity.

Regarding the training procedure, we would like to emphasize that the population-optimal readout is not a particularly restrictive assumption. Once the attention matrix is learned, Stage II reduces to a one-dimensional regression problem, requiring only the estimation of a scalar function. We consider the limit N′→∞N'\to\infty with N′/N→0N'/N\to0, so the readout can be learned using a negligible fraction of additional training samples. In Appendix C.6 (Figure 9), we further show that varying the number of Stage-II prompts has little effect on the memorization and generalization errors. In Appendix C.5 (Figure 8), we provide the comparison where the kernel and the feature learner share the same number of training samples, and the results remain qualitatively the same.

We agree that the two-stage procedure is an idealization of end-to-end training. To assess the effect of the training protocol, we jointly train the attention matrix and nonlinear readout from scratch on the combined Stage-I and Stage-II prompts. The two protocols give similar errors across task-pool sizes (Appendix C.2, Figure 6), and approximately recover the phase diagram (Appendix C.7, Figure 12). The end-to-end training only slightly shifts the crossover toward shorter inference contexts.

Regarding model capacity, we emphasize that the feature learner's advantage cannot be explained simply by having more expressive power or by avoiding overfitting. In fact, the kernel learner has substantially more trainable parameters (D2D^2 versus d2d^2) and employs a rich nonlinear feature representation. For the quadratic teacher used in our experiments, the kernel feature map contains all relevant harmonic components, so there is no representation error. Nevertheless, the feature learner can achieve strictly lower error even in the infinite-pretraining and infinite-task-diversity limit, where overfitting is absent.

Thus, despite the kernel learner's larger attention matrix and complete representation, it performs worse in this regime even with unlimited pretraining data. This demonstrates that the performance gap is not merely a consequence of model capacity or finite-sample overfitting, but reflects a fundamental difference in how the two architectures exploit information from a finite context.

Reviewer_a9fBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies nonlinear in-context learning by comparing two one-layer attention architectures on single-index regression tasks. The kernel learner applies a fixed nonlinear feature map before linear attention, whereas the feature learner applies linear attention to the original inputs followed by a learned nonlinear readout.

The authors derive predictions for memorization and generalization errors using the replica method, accounting for pretraining sample size, task diversity, and training and inference context lengths. The theoretical predictions are compared with numerical experiments, and phase diagrams are provided to characterize regimes in which each architecture performs better.

The main findings suggest that the feature learner generally requires fewer pretraining samples and achieves better generalization in many regimes, while the kernel learner can achieve lower memorization error when task diversity is limited and sufficient pretraining data is available. The paper also establishes different context-length scaling behaviors for the two learners.

Critical strengths
  1. Interesting theoretical perspective. The paper investigates an important question regarding how architectural choices affect nonlinear in-context learning. Comparing kernel and feature learners under a common task distribution provides useful insights into their different memorization and generalization behaviors. The analysis of how pretraining size, task diversity, and context length jointly influence performance is particularly interesting.

  2. Detailed theoretical and empirical analysis. The paper derives quantitative predictions using the replica method and provides numerical experiments supporting these predictions. The phase diagrams help illustrate the regimes in which each architecture is advantageous. In addition, the rigorous characterization of context-length scaling provides a useful theoretical explanation for why the two learners exhibit different behaviors when learning nonlinear targets.

Critical weaknesses
  1. The analysis relies on restrictive modeling assumptions. The theoretical framework focuses on single-index regression tasks, one-layer linear attention, and fixed spherical-harmonic feature maps. Moreover, the finite-pretraining predictions rely on the replica method and a Gaussian-equivalence ansatz, rather than rigorous guarantees. Although these assumptions make the analysis tractable, it remains unclear to what extent the conclusions extend to more general nonlinear tasks, practical feature representations, or standard Transformer architectures.

  2. The architectural comparison is affected by differences in model capacity and training procedures. The kernel and feature learners operate in different feature dimensions, resulting in substantially different numbers of trainable attention parameters. Furthermore, the feature learner uses a two-stage training procedure with an additional learned nonlinear readout, whose theoretical analysis assumes access to sufficient independent training prompts. These differences make it difficult to isolate the effects of nonlinear transformation placement from those of model capacity and training procedures. Additional controlled comparisons would strengthen the interpretation of the observed architectural advantages.

Questions for authors
  1. Regarding the architectural comparison: The kernel and feature learners differ not only in the placement of the nonlinear transformation but also in their feature dimensions, parameter counts, and training procedures. Can the authors clarify to what extent the observed performance differences are caused by the placement of nonlinearity rather than these other factors? Would a comparison with more closely matched model capacities or training budgets lead to similar conclusions?

  2. Regarding the replica-based predictions: The finite-pretraining results rely on the replica method and a Gaussian-equivalence ansatz. While the numerical experiments show good agreement with the theoretical predictions, could the authors clarify the expected limitations of these approximations? In particular, how robust are the predicted phase boundaries to finite-dimensional effects and changes in the feature map or teacher function?

  3. Regarding the two-stage training procedure: The theoretical analysis of the feature learner assumes that the nonlinear readout can be learned with negligible estimation error using additional prompts. Could the authors further investigate how the results depend on the number of samples available for this second stage? In addition, the context-length mismatch phase diagrams appear to rely primarily on theoretical predictions. Can the authors provide direct numerical validation of these regimes?

4 comments

(W1) The analysis relies on restrictive modeling assumptions. The theoretical framework focuses on single-index regression tasks, one-layer linear attention, and fixed spherical-harmonic feature maps. Moreover, the finite-pretraining predictions rely on the replica method and a Gaussian-equivalence ansatz, rather than rigorous guarantees. Although these assumptions make the analysis tractable, it remains unclear to what extent the conclusions extend to more general nonlinear tasks, practical feature representations, or standard Transformer architectures.

Reply to W1: We agree that our analysis focuses on simplified attention architectures. However, such controlled settings are standard in the theoretical study of ICL. For example, Lu et al. (2025) and Takanami et al. (2026) study one-layer linear attention on linear regression tasks. Our work builds on this established framework and extends the analysis to nonlinear single-index regression. This extension reveals a qualitatively new architectural distinction: kernel learners recover different harmonic components at different context-length scales. This separation is rigorously established for a general class of nonlinear teacher functions. The simplified architectures allow us to isolate and characterize how the placement of nonlinearity relative to attention affects nonlinear ICL.

Regarding the non-rigorous approximations, we emphasize that Theorem 1 is proved independently of the replica method and Gaussian equivalence, and agrees with the corresponding limit of the replica predictions. In the revision, we have also extended it to arbitrary training and inference context lengths. We provide extensive numerical validation for the finite-sample predictions.

Extending the analysis to more realistic architectures is an important direction for future work. In particular, the replica framework could potentially be extended to softmax attention, but this introduces additional technical challenges.

(W2) The architectural comparison is affected by differences in model capacity and training procedures. The kernel and feature learners operate in different feature dimensions, resulting in substantially different numbers of trainable attention parameters. Furthermore, the feature learner uses a two-stage training procedure with an additional learned nonlinear readout, whose theoretical analysis assumes access to sufficient independent training prompts. These differences make it difficult to isolate the effects of nonlinear transformation placement from those of model capacity and training procedures. Additional controlled comparisons would strengthen the interpretation of the observed architectural advantages.

Reply to W2: Regarding model capacity, we agree that the different numbers of attention parameters (d2d^2 for the feature learner and D2D^2 for the kernel learner) contribute to their different finite-pretraining sample complexities. However, the feature learner's advantage cannot be explained simply by model capacity or finite-sample overfitting. In fact, the kernel learner has substantially more attention parameters and employs a rich nonlinear feature representation. For the teacher used in our experiments (Figures 2,3), the kernel feature map contains all relevant harmonic components, so there is no representation error.

Nevertheless, the feature learner achieves strictly lower error even in the infinite-pretraining and infinite-task-diversity limit, where overfitting to a finite pretraining dataset or task pool is absent. Thus, despite the kernel learner's larger attention matrix and complete nonlinear feature representation, it performs worse in this regime even with unlimited pretraining data. This demonstrates that the performance gap is not merely a consequence of model capacity or finite-sample overfitting, but reflects a fundamental difference in how the two architectures exploit information from a finite context.

Regarding the training procedure, we emphasize that the population-optimal readout is not a particularly restrictive assumption in our asymptotic setting. Once the attention matrix is learned, Stage II reduces to a one-dimensional regression problem. We consider the limit N′→∞N'\to\infty with N′/N→0N'/N\to0, so the readout can be consistently estimated using an asymptotically negligible fraction of additional training samples. In Figure 8 we further compare the two learners with identical training dataset.

We agree that the two-stage procedure is an idealization of end-to-end training. To assess the effect of the training protocol, we jointly train the attention matrix and nonlinear readout from scratch on the combined Stage-I and Stage-II prompts. The two protocols give similar errors across task-pool sizes (Appendix C.2, Figure 6), and approximately recover the phase diagram (Appendix C.7, Figure 12). The end-to-end training only slightly shifts the crossover toward shorter inference contexts.

(Q1) Regarding the architectural comparison: The kernel and feature learners differ not only in the placement of the nonlinear transformation but also in their feature dimensions, parameter counts, and training procedures. Can the authors clarify to what extent the observed performance differences are caused by the placement of nonlinearity rather than these other factors? Would a comparison with more closely matched model capacities or training budgets lead to similar conclusions?

Reply to Q1: As discussed in our response to W2, we acknowledge that differences in model capacity contribute to the finite-pretraining sample efficiency of the two learners. However, our rigorous result demonstrates that the feature learner's advantage persists even with unlimited pretraining data, despite the kernel learner having more attention parameters and a complete representation. This rules out finite-sample overfitting as the explanation for the separation and identifies the placement of nonlinearity as a key architectural factor.

We also match the training budget by controlling the total number of training data in Figure 8 (Appendix C.5). We find that the feature and kernel learners retain the same ordering, for both memorization and generalization.

(Q2) Regarding the replica-based predictions: The finite-pretraining results rely on the replica method and a Gaussian-equivalence ansatz. While the numerical experiments show good agreement with the theoretical predictions, could the authors clarify the expected limitations of these approximations? In particular, how robust are the predicted phase boundaries to finite-dimensional effects and changes in the feature map or teacher function?

Reply to Q2: Our replica predictions are supported by their agreement with our independent rigorous results, and numerical experiments. Regarding the limitations of these approximations, we distinguish two aspects. First, the replica-symmetric ansatz is particularly natural for the convex optimization problems considered in our main experiments, but its validity for general nonlinear Stage-I readouts, which may lead to nonconvex optimization, is less clear. Second, Gaussian equivalence remains unproved for our structured ICL data. Establishing universality for these correlated features is the main technical challenge in rigorously justifying the predictions.

Nevertheless, we expect the predictions to remain asymptotically valid in the high-dimensional regimes considered in our analysis with positive regularization, including near interpolation thresholds. The deviations observed near the kernel interpolation peak in Appendix C.1 may reflect larger finite-size corrections rather than a breakdown of the asymptotic predictions.

To assess finite-dimensional effects, we compare the experimental phase diagrams at d=32 and d=24 (Appendix C.7, Figures 10 and 11) with the replica predictions in Figure 3. Both figures recover the predicted phase diagram, with slight shifts in the boundaries. This supports the robustness of the replica phase diagrams at finite dimensions.

Regarding the teacher function, our theoretical framework is not restricted to the quadratic teacher used in the experiments, and Theorem 1 applies to a broad class of nonlinear functions. While the precise phase boundaries naturally depend on the teacher's Hermite coefficients, the qualitative separation in context-length scaling remains valid under the assumptions of our theorem. To assess sensitivity to the teacher function, we consider a sinusoidal teacher that has higher-order Hermite components in Figure 13. The replica predictions agree well with the numerical results for the sinusoidal teacher, supporting the applicability of our theory beyond the quadratic teacher.

Regarding the feature map, our analysis allows different collections of spherical-harmonic degrees, but does not establish universality for arbitrary nonlinear embeddings. Extending the quantitative predictions to random features or other practical embeddings is an interesting direction for future work.

(Q3) Regarding the two-stage training procedure: The theoretical analysis of the feature learner assumes that the nonlinear readout can be learned with negligible estimation error using additional prompts. Could the authors further investigate how the results depend on the number of samples available for this second stage? In addition, the context-length mismatch phase diagrams appear to rely primarily on theoretical predictions. Can the authors provide direct numerical validation of these regimes?

Reply to Q3: Regarding the training procedure, we would like to emphasize that the population-optimal readout is not a particularly restrictive assumption. Once the attention matrix is learned, Stage II reduces to a one-dimensional regression problem, requiring only the estimation of a scalar function. We consider the limit N′→∞N'\to\infty with N′/N→0N'/N\to0, so the readout can be learned using an negligible fraction of additional training samples. In Appendix C.6, we further show that the number of samples for the second stage has a negligible influence on the results.

Regarding context-length mismatch, we have substantially strengthened Theorem 1 to rigorously characterize the population errors for arbitrary training and inference context lengths LL and L~\widetilde L, which provides a rigorous explanation for the context-length mismatch mechanisms underlying Figures 3(c,d), independently of the replica method and Gaussian-equivalence assumptions.

In Figures 10(c,d) and 11(c,d) in Appendix C.7, we directly test the context-length mismatch predictions in Figure 3(c,d) at d=32 and d=24, respectively. At both dimensions, the experiments recover the predicted phase diagram.

Reviewer_a9fBOpenReview ↗

Reply to authors

Thank you for the additional experiments. Matching the total number of training prompts and testing end-to-end training substantially address my concerns about the two-stage protocol. The experiments at d=24d=24 and d=32d=32 also support the main trends in the context-length mismatch phase diagrams. I find the population-level scaling result convincing. My remaining reservation concerns the finite-pretraining comparison, where the different parameter counts make it difficult to isolate the effect of nonlinearity placement. Overall, I maintain my weak acceptance rating.

Comments on the paper

1 comment

We thank all reviewers for their constructive comments. We have substantially revised the manuscript to address the main concerns. All changes in the revised manuscript are highlighted in blue. The key changes are summarized below.

  1. Strengthened rigorous theoretical results. We extended Theorem 1 to characterize the population errors for different training and inference context lengths LL and L~\widetilde L. This provides a rigorous explanation of context-length mismatch, independently of the replica method and Gaussian-equivalence assumptions.

  2. Direct validation of the phase diagrams. We added numerical validation of the phase diagrams at d=24 and d=32, including experiments with mismatched context lengths (Figures 10–11). We also tested the phase diagrams under end-to-end training (Figure 12).

  3. Additional robustness experiments. We considered a sinusoidal teacher with higher-order Hermite components (Figure 13), different regularization strengths (Figure 7), and added error bars from three independent runs to the main experiments (Figures 2, 5, and 6).

  4. More controlled comparisons. We compared the learners under equal total training-sample budgets (Figure 8), investigated the sensitivity to Stage-II sample size (Figure 9), and evaluated end-to-end training against the two-stage procedure (Figures 6 and 12).

Overall, these additions strengthen both the theoretical foundations and empirical support for our conclusions. Further details on these revisions and how they address the reviewers' concerns are provided in our individual responses below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.