acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysis

Abstract

Prior Fitted Networks (PFNs) such as TabPFN zhang2025tabpfnonemodel now rival established statistical procedures across prediction and estimation tasks. A natural explanation is that PFNs have the property of statistical adaptivity, that is they perform nearly as well as a method tailored to the true data-generating model for a heterogeneous set of models, while not being told which model the data comes from. Prior work does not give a conclusive explanation, how gradient-based pretraining learns an adaptive estimator. We study how such adaptivity is learned in a controlled location-estimation problem. Each task is an unlabeled sample whose family is hidden: Gaussian data call for averaging, with error of order , whereas uniform data are best estimated from their extremes, at the faster rate . We also provide the example of a symmetric Gaussian mixture, for which a rate of can be attained. On scalar inputs, softmax attention computes the derivative of the empirical cumulant-generating function. A single primitive therefore both supplies features that distinguish the families and forms estimators interpolating between the sample mean and the mid-range. Building on this, we combine attention experts through either a softmax mixture of experts or a gated linear unit (GLU), and analyze stagewise gradient flow. With pretraining tasks, the learned estimator is asymptotically efficient on Gaussian tasks, within a factor of the minimax rate on uniform tasks, and order-optimal on mixtures in a shrinking-variance regime. These guarantees extend to new locations and longer contexts. A risk decomposition separates expert error, routing error and normalization error, which clarifies the architectural contrast. Softmax gating enforces normalization and exact translation equivariance, whereas the GLU must learn it: its dynamics separate into fast bias removal followed by slow expert selection. End-to-end experiments recover the predicted specialization.

Primary area:
learning theory
Keywords:
in-context learning, adaptivity, mean estimation, learning dynamics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aZwhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether pretraining can learn to perform adaptive statistical estimation: given an unordered set of n scalar observations, estimate the shared location without being told which distribution generated them. The two candidate families are Gaussian versus uniform, or Gaussian versus a symmetric two-component Gaussian mixture — not all three at once, and not arbitrary distributions.

Critical strengths

The statistical problem is well chosen.

The mechanism is interpretable.

The appendices contain real checks.

The paper is careful about what equivariance does and does not give.

Critical weaknesses

Two prescribed candidate shapes, fixed observation scale, one-dimensional target, pre-assigned estimator and feature heads, paired opposite scales, and staged training. It explains how this mechanism learns to adapt; it does not yet explain how unconstrained transformers or tabular PFNs acquire broad adaptivity.

The theorem samples expert scales independently from [−1, 1]; the joint experiments draw from [0, 1.9] and reject separations below 0.6, and the stagewise experiments inherit this. Post-hoc relabeling of experts is legitimate, but failure rates under the theorem's own initialization are not reported.

Feature separation is positive almost surely, but constants depend on the realized feature draw, on initial expert covariance gaps, and on 1/λ_min(Z). Near-collinear features or near-tied initial experts plausibly make the required sample size and training time very large; the paper should state this dependence on n and η explicitly.

Questions for authors

See weakness

2 comments

Thank you for highlighting the choice of statistical problem, the interpretability of the mechanism, the checks in the appendices, and the discussion of what equivariance guarantees. Below, we address the scope of the result, the initialization used in the experiments, and the effect of feature separation and covariance gaps on convergence.

W1. Architecture and adaptivity

The theory isolates one mechanism through which gradient flow learns adaptivity. We agree that our results do not explain how unconstrained transformers acquire broad adaptivity. Analyzing the training dynamics requires trading off the complexity of the architecture against the generality of the result (see our response to Reviewer Dp53, Q). Since the components are standard, we expect the mechanism to carry over to larger architectures (see our response to Reviewer 8kqn, Q3). The experiments remove the prescribed head roles and paired scales, train all parameters jointly, and include four families. The scalar target and the fixed observation scale remain.

Experiments support the proposed mechanism in broader architectures. In the revision, we add a two-layer transformer with no fixed parameters, trained on Gaussian, uniform, Gaussian-mixture, and Laplace tasks (Appendix I.6). It exhibits fast bias removal followed by slower family-specific improvements (Figure 15), consistent with our analysis, and several gates of both GLUs take different values on the four families (Figure 16).

W2. Dependence on feature separation and initialization

Poorly separated features or nearly tied initial experts can indeed make the required sample size and training time large.

Extended theory. We added Corollary E.3, which makes the dependence on the initial expert parameters explicit for the softmax MoE. Conditional on a fixed feature draw and initialization satisfying the corollary's assumptions, including bounded initial gating parameters, the following sample-size and training-time bounds guarantee a gate error of at most 2/n2/n:

n\ge C|λ_1(0)|^{-64/(ϱϵ)},\quad t_\theta \le \frac{Cn^2}{|λ_1(0)|^2(|λ_2(0)|^2-|λ_1(0)|^2)}.$$ Since $|λ_k(0)|\sim\mathrm{Unif}[0,1]$, with probability at least $1-\eta$ over the expert initialization we have $|λ_1(0)|\ge\eta/4$ and $|λ_2(0)|-|λ_1(0)|\ge\eta/4$, so the bounds become $n\ge C(4/\eta)^{64/(ϱϵ)}$ and $t_\theta\le Cn^2/\eta^4$ (Corollary E.3). For the GLU, we do not have an explicit bound of this form. Since the same covariance gaps drive its expert-selection phase, we expect a similar dependence. For empirical gradient flow, this guarantee holds with probability at least $1-1/n$ over the training tasks, provided $m\ge n^{1+ϵ}$ and $n\ge C(|λ_1(0)|(|λ_2(0)|^2-|λ_1(0)|^2))^{-64/(ϱϵ)}$, which becomes $n\ge C(64/\eta^3)^{64/(ϱϵ)}$ with probability at least $1-\eta$ over the expert initialization. These bounds show how the sample size and training time required for the guarantee depend on the initial expert parameters and feature separation, explaining why nearly tied initial experts or poorly separated features can require larger training budgets. **Revised experiments.** We reran all joint and stagewise experiments with the initialization of the theorem, without rejection. Since changing the sign of $λ_k$ leaves the expert unchanged, drawing $|λ_k(0)|$ uniformly from $[0,1]$ is equivalent to drawing $λ_k(0)$ uniformly from $[-1,1]$. We previously sampled from $[0, 1.9]$, since numerical checks indicate that the expert hierarchy holds on this range. Our proof covers $[0, 1]$. We previously rejected separations below $0.6$ to avoid slow convergence due to bad initialization. After relabeling the experts, 31 of 32 joint runs and 15 of 20 stagewise GLU runs are successful. All failed runs start with $\min\{|λ_1(0)|,|λ_2(0)|-|λ_1(0)|\}<0.07$, in line with the dependence on the initial expert parameters in Corollary E.3, and every run with a larger value is successful. ### W3. Feature draw **The feature draw enters the bounds through its separation.** The feature draw does affect the speed of convergence. In Corollary E.3, it enters through $ϱ$, a lower bound on $λ_{\min}(Z)=1-\langle\bar{\mathbf z}_{\mathrm G},\bar{\mathbf z}_{\mathrm A}\rangle$, and through the constant $C$. As $d$ grows, the law of large numbers drives $\langle\bar{\mathbf z}_{\mathrm G},\bar{\mathbf z}_{\mathrm A}\rangle$ to the correlation of $m_{\mathrm G}(\omega)$ and $m_{\mathrm A}(\omega)$ under $\omega\sim\mathcal N(0,1)$, so for large $d$ a typical draw has separation close to a fixed positive constant. We therefore only made the dependence on the initial expert parameters explicit. In our reruns, this dependence accounts for all failures, since every failed run starts with nearly vanishing or nearly tied expert scales.

No more questions, and keep my positive rating.

Reviewer_8kqnOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

In this work, the authors show that a small attention-based model can learn, through gradient-based training, to estimate the center of a dataset adaptively, without being told the data’s distribution. The key idea is that attention computes a weighted average controlled by one parameter, which can produce both estimators and also helps identify the distribution. The authors prove that, under a two-stage training procedure, their estimator matches the sample mean on Gaussian data, nearly matches the more accurate mid-range on uniform data, and achieves the optimal order on a Gaussian mixture. Experiments illustrate this behavior.

Critical strengths
  • Clear, simple insight

showing that a single attention module can act both as an estimator for the center and as a feature for identifying the distribution is elegant and easy to understand. On top the authors proof that gradient-based training actually finds a good solution.

  • Meaningful architecture comparison

the contrast between softmax gating, which enforces normalization automatically, and GLU gating, which must learn normalization in a fast phase before slowly learning which expert to select, is insightful beyond the specific estimation problem studied, since both gating mechanisms are widely used in standard transformer architectures.

Critical weaknesses
  • Simplified architecture

The fact that the architecture seems somewhat tailored towards this specific problem makes it unclear to me how much of the adaptivve behavior is actually learned and how much is built in by design. Many components are fixed in advance rather than trained, including the split into expert and feature heads as well as the feature parameters.

  • Limited scope

Why does the analysis consider only two distribution families at a time? Could the approach extend to more families, for example by adding more experts? And since the estimator can only interpolate between the mean and the mid-range, how would it handle distributions where other estimators are needed, such as heavy-tailed data where median-type estimators are preferable?

  • Transfer to larger models

It is not clear to me how these findings translate to larger models such as TabPFN. Can the authors please clarify whether they expect the same mechanism, attention acting both as an estimator and as a way to identify the distribution, to emerge in larger models trained on more diverse data, and whether this could be tested empirically?

Questions for authors
  • How much of the adaptive behavior depends on the fixed components? For example, does the model still learn to distinguish the families if the split into expert and feature heads, or the feature parameters, are trained rather than fixed?

  • Could the analysis be extended to more than two distribution families, for example by using more experts, and what would be the main obstacle?

  • Do the authors expect the same mechanism to appear in larger pretained models such as TabPFN, and is there an experiment that could test this?

3 comments

Thank you for highlighting the dual role of attention in estimation and distribution identification, and the contrast between softmax and GLU gating. We appreciate your questions about which parts of the adaptive behavior are learned and how the approach might extend to more distributions and larger models. We also revised the abstract, Section 2.1, and the opening of Section 3 to improve readability (see our response to Reviewer Dp53).

Q1.1. How much of the adaptive behavior depends on the fixed components?

Fixing components removes symmetries and leaves a nontrivial learning problem. A standard transformer has many symmetries such as permutations of heads, and the same heads can serve both as estimators and as gating features, so many parametrizations implement the same estimator. Fixing which heads estimate and which identify the family removes these symmetries and makes the gradient flow tractable. The learning problem remains nontrivial because the gating dynamics have suboptimal stationary points at which the gate selects the wrong expert for one or both families, and our analysis shows that gradient flow avoids them (Appendix D).

Q1.2. For example, does the model still learn to distinguish the families if the split into expert and feature heads, or the feature parameters, are trained rather than fixed?

Yes. In both experiments below, no heads are designated as expert or feature heads, all attention parameters are trained, and the model still distinguishes the families.

  1. Relaxed architecture (Appendix I.5). We train the base architecture of Definition 2.2 with the GLU and without LayerNorm, so the same trained attention outputs serve as gating features and as inputs to the experts. Its Gaussian error stays near the sample-mean error, and its uniform error falls well below it toward the sample mid-range error (Figure 13). One gate takes clearly different values on Gaussian and uniform contexts (Figure 14), so the family is identified from learned features.
  2. Two-layer transformer (Appendix I.6). With no fixed parameters and four families, the Gaussian error stays near the sample-mean error, and the errors on the other three families fall well below it (Figure 15). Several gates of both GLUs take different values on the four families (Figure 16).

Q2. Could the analysis be extended to more than two distribution families, for example by using more experts, and what would be the main obstacle?

Theoretical tractability. Adding experts lets the architecture accommodate more families, provided some expert attains the required accuracy on each family. The main obstacle is the convergence proof for the gating dynamics. For two experts and two families, the covariance ordering of the experts and the geometry of the two family features exclude convergence to suboptimal stationary points. With more families, the gate must separate several feature directions at once, and both conditions and the avoidance argument would need to be generalized. Empirically, the two-layer transformer of Appendix I.6 adapts to four families (Figure 15), which is not covered by our theory. Conceptual framing. Constructions such as Bai et al. (2023) show that transformers can implement in-context algorithm selection for general tasks. We instead analyze how gradient flow learns such a mechanism, which requires a more restricted setting. Our analysis provides a proof of concept for how prior-fitted networks learn adaptive estimation.

Q3. Do the authors expect the same mechanism to appear in larger pretrained models such as TabPFN, and is there an experiment that could test this?

We expect the mechanism to carry over, because it relies only on standard components. On scalar observations, softmax attention computes exponentially tilted averages, which can both identify the distribution of a sample and form estimates. Such column-wise attention pooling is used in TabICL (see our response to Reviewer Dp53). A gated linear unit can then use the representation to gate between estimates based on family-sensitive features. The two-layer transformer of Appendix I.6, trained end to end without prescribed heads, shows the two signatures predicted by our analysis: several gates of both GLUs separate the four families (Figure 16), and the response to location shifts is learned before the family-specific improvements (Figure 15). The proposed mechanism suggests concrete tests for large models. Tracking the bias and variance of the predictions during training, and probing GLU activations across families, would test for these signatures in models such as TabPFN. Performing these checks on models of the size of TabPFN is beyond the scope of our paper.

W2. Heavy tails and median-type estimators

The transformer of Appendix I.6 is now also trained on Laplace data, for which the sample median is efficient. In our stylized architecture, each expert interpolates between the sample mean and the mid-range, so median-type estimators lie outside the analyzed class. For Laplace data with unit variance, the Cramér–Rao bound is 1/(2n)1/(2n), which the sample median attains asymptotically. The sample mean has risk 1/n1/n. The trained transformer reduces the Laplace error to about 0.6/n0.6/n, well below the sample mean and close to the bound (Figure 15).

Thank you for your response. I will keep my positive rating.

Reviewer_Dp53OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies adaptive location estimation through in-context learning. The authors introduce an architecture inspired by multi-head attention and mixture-of-experts models, and analyze its gradient-flow dynamics when trained on tasks drawn from Gaussian, uniform, and Gaussian-mixture distributions. They show that the learned parameters converge to estimators that adapt to the underlying distribution, achieving asymptotically optimal rates for Gaussian data, nearly optimal rates (up to an (n^\epsilon) factor) for uniform data, and order-optimal rates for Gaussian mixtures under additional assumptions.

Critical strengths

I think this is an interesting theoretical result. The analysis of gradient-flow dynamics and how the architecture learns to select different estimators depending on the input distribution is nontrivial. The resulting near-optimal statistical rates are also interesting.

Critical weaknesses

My main concern is that the architecture seems somewhat artificial and specifically designed for this problem, rather than resembling an actual transformer. It is unclear what these results tell us about the dynamics of in-context learning in more realistic architectures. I did not find sufficient justification for this connection, and I would appreciate further discussion or evidence from the authors.

There are also some presentation issues. For example, the discussion of Gaussian and uniform estimation rates in the abstract feels disconnected from the surrounding motivation, and Section 3 starts directly with the main theorem without any introduction. Finally, the role and justification of the initialization ordering (|\lambda_1(0)|<|\lambda_2(0)|) should be explained more clearly.

Overall, I find the theoretical results interesting, but the motivation from the perspective of in-context learning is not yet convincing to me. A stronger justification of the architecture and its relevance to transformer training dynamics would improve my assessment.

Questions for authors

What does this result inform us about the dynamics of in-context learning in actual transformer architecture? (even in toy settings)

3 comments

Thank you for recognizing the gradient-flow analysis and the near-optimal estimation rates. Your central concern is whether the analyzed architecture provides insight into in-context learning in less constrained transformers. We agree that this connection needs to be supported explicitly. The revised draft now includes an expanded experiment with the relaxed architecture (Appendix I.5) and a new two-layer transformer experiment, including measurements of its training dynamics.

W1. Architecture and relation to transformers

Adaptive estimation persists when the prescribed head roles are removed. Appendix I.5 studies the broader architecture in Definition 2.2, in which the same ten attention outputs feed both the candidate estimates and the gate. This removes the separation into estimation and routing heads, the paired expert scales, and the centering and normalization of the routing features. The attention scales and GLU projection matrices are trained jointly. The final readout is fixed to summation, W3=1⊤W_3=\mathbf{1}^{\top}. Across ten independent runs, Gaussian validation error stays near the sample-mean error, and uniform error falls well below it toward the sample mid-range error (Figure 13). Figure 14 additionally shows family-dependent gate values in one run.

A two-layer transformer also learns distribution-dependent behavior. In the new Appendix I.6, each scalar observation is embedded as a token, a learned query token is prepended, and two layers apply multi-head attention followed by a GLU, with LayerNorm and residual connections. The prediction is a linear readout of the query token. All parameters are trained jointly, with no prescribed expert or routing heads.

We train this model on Gaussian, uniform, Gaussian-mixture, and Laplace tasks, with n=70n=70 observations per context. Gaussian error remains near the sample-mean error, and the errors on the other three families fall below it toward their respective references (Figure 15). Several gates in both layers take different values across families (Figure 16). For example, the family explains 95%95\% of the variation of one first-layer gate. This experiment uses one training run and 2,000 validation tasks per family.

The architectural connection is through attention pooling. In our scalar head, a readout query attends to observations as keys and values, with scores λxi\lambda x_i, producing

h(x,λ)=∑ixieλxi∑ieλxi.h(\mathbf{x},\lambda)=\frac{\sum_i x_i e^{\lambda x_i}}{\sum_i e^{\lambda x_i}}.

Related aggregation appears in TabICL's column encoder, where inducing queries attend across observations to summarize distributional information (TabICL, Section 3.2; TabICLv2, Appendix A.2). Our analysis additionally imposes separate head roles, paired expert scales, fixed routing-head scales, and stagewise training. Section 2.1 now introduces the transformer architecture (Definition 2.2) before specifying these restrictions, making their role explicit.

Q. What does the result tell us about ICL dynamics in actual transformers, even in toy settings?

The theorem establishes a mechanism through which pretraining learns statistical adaptation from prediction loss alone. During pretraining, the learner observes contexts and target locations, without family labels or prescribed estimator choices. At inference, the trained parameters are fixed and the context determines the estimate. The analysis explains how gradient-based pretraining learns this context-dependent procedure.

Two findings are particularly relevant to the transformer experiments:

  1. Attention can supply both estimates and evidence for choosing between them. At zero tilt, the scalar attention output gives the sample mean and at large tilts, it approaches the sample maximum and minimum. Fixed-tilt evaluations also supply features that distinguish the families. Lemma 3.3 shows how differences in the initial experts' errors drive the gate toward family-dependent selection through the location loss. Lemma 3.4 then shows how the selected experts specialize. Training learns the distribution-dependent choice and the expert parameters.

  2. Learning the correct response to location shifts can precede learning the appropriate estimator. For the analyzed GLU, Equation (7) separates a normalization-error term at scale σμ2\sigma_\mu^2 from an expert-variance term at scale 1/n1/n. Their different scales lead to fast approximate normalization followed by slower expert selection. Softmax gating enforces normalization and exact translation equivariance by construction.

    In the two-layer transformer (Appendix I.6), the corresponding bias falls to about 10−310^{-3} within 400 steps, while the non-Gaussian errors keep improving (Figure 15).

The gate measurements and temporal ordering do not establish that the two-layer transformer implements the same expert decomposition or obeys the same gradient-flow equations. Proving that connection for jointly trained multilayer transformers remains open.

W2. Initialization ordering

The ordering ∣λ1(0)∣<∣λ2(0)∣|\lambda_1(0)|<|\lambda_2(0)| is a labeling convention. Definition 2.3 initializes the expert scales independently from Unif[−1,1]\mathrm{Unif}[-1,1], so their magnitudes are distinct almost surely. Because the expert roles are exchangeable, Theorem 3.1 relabels the experts and permutes the corresponding gating parameters together. No initialization is selected or rejected on the basis of this ordering.

What matters is the covariance ordering within the analyzed initialization range. The smaller-magnitude expert is favored on Gaussian contexts and the larger-magnitude expert on the alternative. This supplies the initial learning signal for the gate. The learner receives neither family labels nor the desired assignment. Section 3 now explains the relabeling before Theorem 3.1, and Section 3.2 describes the covariance ordering and its role in the gating dynamics.

W3. Presentation

We have addressed both presentation concerns. The abstract now explains the statistical motivation before giving the rates. Section 3 now opens with an overview of the main risk guarantees, fixed-location transfer, and longer-context generalization before stating Theorem 3.1. It also clarifies the initialization ordering at that point.

Dear Reviewer Dp53,

Thank you again for taking the time to review our paper. We have uploaded a revised manuscript and posted our response to your comments.

Before the discussion period closes, we would greatly appreciate it if you could let us know whether our revisions and responses have adequately addressed your concerns. We would be happy to provide further clarification on any remaining points.

Best,

The Authors

Comments on the paper

1 comment

We thank the reviewers for their thoughtful comments.

Strengths mentioned by the reviewers

Dp53: The gradient-flow analysis is nontrivial and the near-optimal rates are interesting.

8kqn: Clear insight that attention acts both as an estimator and as a feature for identifying the distribution, and a meaningful comparison of softmax and GLU gating.

aZwh: The paper gives an interpretable mechanism, the statistical problem was well chosen, and the appendices contain real checks.

Responses to the reviewers’ concerns

Dp53: (Relation to transformers) The simplifications make the gradient flow tractable, and a two-layer transformer with no fixed parameters shows dynamics consistent with our analysis.

8kqn: (Fixed components, more families) Without fixed components, the relaxed architecture and a two-layer transformer still distinguish the families, the latter across four families including the Laplace.

aZwh: (Dependence on initialization) Corollary E.3 makes the dependence on the initial expert parameters explicit and experiments support this finding.

Manuscript Updates (changes are highlighted in blue)

Main Text The abstract, Section 2.1 (Definition 2.2), and the opening of Section 3 were revised to improve presentation and clarify the relation to transformer architectures. Section 3.2 states the explicit gating bound of Corollary E.3, and the Discussion covers the two-layer transformer experiment.

Appendix

  1. Corollary E.3 was added, addressing the concern of Reviewer aZwh regarding the dependence on the initial expert parameters.
  2. The relaxed-architecture experiment was expanded (Appendix I.5), and a two-layer transformer experiment on Gaussian, uniform, Gaussian-mixture, and Laplace tasks was added (Appendix I.6).
  3. All joint and stagewise experiments were rerun with the initialization of Theorem 3.1, without rejection, addressing the concern of Reviewer aZwh; success rates are reported.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.