- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies whether pretraining can learn to perform adaptive statistical estimation: given an unordered set of n scalar observations, estimate the shared location without being told which distribution generated them. The two candidate families are Gaussian versus uniform, or Gaussian versus a symmetric two-component Gaussian mixture — not all three at once, and not arbitrary distributions.
Critical strengths
The statistical problem is well chosen.
The mechanism is interpretable.
The appendices contain real checks.
The paper is careful about what equivariance does and does not give.
Critical weaknesses
Two prescribed candidate shapes, fixed observation scale, one-dimensional target, pre-assigned estimator and feature heads, paired opposite scales, and staged training. It explains how this mechanism learns to adapt; it does not yet explain how unconstrained transformers or tabular PFNs acquire broad adaptivity.
The theorem samples expert scales independently from [−1, 1]; the joint experiments draw from [0, 1.9] and reject separations below 0.6, and the stagewise experiments inherit this. Post-hoc relabeling of experts is legitimate, but failure rates under the theorem's own initialization are not reported.
Feature separation is positive almost surely, but constants depend on the realized feature draw, on initial expert covariance gaps, and on 1/λ_min(Z). Near-collinear features or near-tied initial experts plausibly make the required sample size and training time very large; the paper should state this dependence on n and η explicitly.
Questions for authors
See weakness
2 comments
No more questions, and keep my positive rating.
Thank you for highlighting the choice of statistical problem, the interpretability of the mechanism, the checks in the appendices, and the discussion of what equivariance guarantees. Below, we address the scope of the result, the initialization used in the experiments, and the effect of feature separation and covariance gaps on convergence.
W1. Architecture and adaptivity
The theory isolates one mechanism through which gradient flow learns adaptivity. We agree that our results do not explain how unconstrained transformers acquire broad adaptivity. Analyzing the training dynamics requires trading off the complexity of the architecture against the generality of the result (see our response to Reviewer Dp53, Q). Since the components are standard, we expect the mechanism to carry over to larger architectures (see our response to Reviewer 8kqn, Q3). The experiments remove the prescribed head roles and paired scales, train all parameters jointly, and include four families. The scalar target and the fixed observation scale remain.
Experiments support the proposed mechanism in broader architectures. In the revision, we add a two-layer transformer with no fixed parameters, trained on Gaussian, uniform, Gaussian-mixture, and Laplace tasks (Appendix I.6). It exhibits fast bias removal followed by slower family-specific improvements (Figure 15), consistent with our analysis, and several gates of both GLUs take different values on the four families (Figure 16).
W2. Dependence on feature separation and initialization
Poorly separated features or nearly tied initial experts can indeed make the required sample size and training time large.
Extended theory. We added Corollary E.3, which makes the dependence on the initial expert parameters explicit for the softmax MoE. Conditional on a fixed feature draw and initialization satisfying the corollary's assumptions, including bounded initial gating parameters, the following sample-size and training-time bounds guarantee a gate error of at most 2/n:
n\ge C|λ_1(0)|^{-64/(ϱϵ)},\quad t_\theta \le \frac{Cn^2}{|λ_1(0)|^2(|λ_2(0)|^2-|λ_1(0)|^2)}.$$ Since $|λ_k(0)|\sim\mathrm{Unif}[0,1]$, with probability at least $1-\eta$ over the expert initialization we have $|λ_1(0)|\ge\eta/4$ and $|λ_2(0)|-|λ_1(0)|\ge\eta/4$, so the bounds become $n\ge C(4/\eta)^{64/(ϱϵ)}$ and $t_\theta\le Cn^2/\eta^4$ (Corollary E.3). For the GLU, we do not have an explicit bound of this form. Since the same covariance gaps drive its expert-selection phase, we expect a similar dependence. For empirical gradient flow, this guarantee holds with probability at least $1-1/n$ over the training tasks, provided $m\ge n^{1+ϵ}$ and $n\ge C(|λ_1(0)|(|λ_2(0)|^2-|λ_1(0)|^2))^{-64/(ϱϵ)}$, which becomes $n\ge C(64/\eta^3)^{64/(ϱϵ)}$ with probability at least $1-\eta$ over the expert initialization. These bounds show how the sample size and training time required for the guarantee depend on the initial expert parameters and feature separation, explaining why nearly tied initial experts or poorly separated features can require larger training budgets. **Revised experiments.** We reran all joint and stagewise experiments with the initialization of the theorem, without rejection. Since changing the sign of $λ_k$ leaves the expert unchanged, drawing $|λ_k(0)|$ uniformly from $[0,1]$ is equivalent to drawing $λ_k(0)$ uniformly from $[-1,1]$. We previously sampled from $[0, 1.9]$, since numerical checks indicate that the expert hierarchy holds on this range. Our proof covers $[0, 1]$. We previously rejected separations below $0.6$ to avoid slow convergence due to bad initialization. After relabeling the experts, 31 of 32 joint runs and 15 of 20 stagewise GLU runs are successful. All failed runs start with $\min\{|λ_1(0)|,|λ_2(0)|-|λ_1(0)|\}<0.07$, in line with the dependence on the initial expert parameters in Corollary E.3, and every run with a larger value is successful. ### W3. Feature draw **The feature draw enters the bounds through its separation.** The feature draw does affect the speed of convergence. In Corollary E.3, it enters through $ϱ$, a lower bound on $λ_{\min}(Z)=1-\langle\bar{\mathbf z}_{\mathrm G},\bar{\mathbf z}_{\mathrm A}\rangle$, and through the constant $C$. As $d$ grows, the law of large numbers drives $\langle\bar{\mathbf z}_{\mathrm G},\bar{\mathbf z}_{\mathrm A}\rangle$ to the correlation of $m_{\mathrm G}(\omega)$ and $m_{\mathrm A}(\omega)$ under $\omega\sim\mathcal N(0,1)$, so for large $d$ a typical draw has separation close to a fixed positive constant. We therefore only made the dependence on the initial expert parameters explicit. In our reruns, this dependence accounts for all failures, since every failed run starts with nearly vanishing or nearly tied expert scales.