acceptodds
Under review as a conference paper at ICLR 2027

Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention

Abstract

Estimating a decision threshold requires locating observations near an unknown boundary. We study how gradient-based pretraining learns this statistical rule in a two-parameter softmax-attention model with a fixed feature and inequality direction. Pretraining uses labeled contexts and their true thresholds; a fresh threshold must be inferred from context alone. Under a large-resolution initialization, constant-step gradient descent on tasks with examples each produces a frozen estimator with error for each fixed interior threshold and every fresh-context size . The two terms separate finite-pretraining accuracy from fresh-context localization. The mechanism is coordinated parameter divergence: population training calibrates the relative label and feature scores, then increases the attention scale as , giving population threshold error . To transfer this mechanism to a fixed finite corpus, we control gradient errors relative to the shrinking directions of progress at successive parameter scales. This certifies a growing training interval without requiring long-time tracking of the population trajectory. We also identify the boundary limitation of the one-head model and explain statistically what a reflected symmetrization could achieve.

Primary area:
learning theory
Keywords:
in-context learning, amortized estimation, softmax attention, gradient descent dynamics, threshold estimation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wj3POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates the theoretical properties of a simple restricted softmax attention model. Specifically, it examines whether such a model can learn to predict the location of a label transition if trained using gradient descent, under narrow data distribution assumptions. The paper proves that under infinite data, gradient descent approaches the optimal solution in the limit. It also establishes that gradient descent achieves bounded estimation error on finite data, with the bound tightening with more training data. The authors then perform synthetic experiments that validate the correctness of the theory.

Critical strengths
  1. Although I am not a theoretical ML researcher, the technical work that went into producing the theory seems substantial.
  2. To the best of my understanding, the theoretical results are correct and accurately capture the behaviour of the restricted softmax attention model in the scenario under investigation.
Critical weaknesses
  1. Unclear practical motivation. As I am not a theoretical ML researcher, I will not place strong judgement on this, as I am unaware of the standards/norms in this research area. However, I am unable to discern how the theoretical results could be of use to practitioners. The paper weakly motivates the research via in-context learning, transformer learning on tabular data and amortised estimation; however, I am struggling to see how the insights produced around the narrow problem scenario investigated port to these motivating problems. The theory pertains to direct supervision of an extremely restricted two-parameter softmax attention model to predict a one-dimensional transition threshold, where the threshold is the explicit training target. As far as I can tell, this has very little to do with practical transformer learning, or even simple one layer neural networks, which use fully parameterised attention and are trained end to end with a task loss. The ICL framing is also weak -- the problem can be simply cast as supervised learning from what I can tell.
  2. Presentation could be improved. As mentioned above, the motivation of the problem is loosely linked to the actual research contribution, making reading the paper quite confusing. Furthermore, although the structure of the paper is laid out at the end of the introduction, it is confusing to follow section to section -- in my opinion, how results depend on each other should be better signposted. The experimental figures are difficult to parse: text is far too small; legends and axis labels use locally unexplained notation; and captions rely heavily on the main body -- they are not understandable in isolation. Good figures should not only state empirical results but also improve/enable better understanding/intuition for the reader.
Questions for authors

Could the authors describe to me a practical scenario in which the theoretical insight in this paper could be of use?

4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Part 1/2

We thank the reviewer for recognizing the substantial technical work and the correctness of our theory. We respond to the weaknesses and question below.

In-context estimation differs from conventional supervised learning

We agree that one can cast our problem as supervised learning, with each dataset as a "training example" and the threshold as its label. Indeed, prior work on ICL for statistical estimation concedes this point, calling it "supervised learning of statistical estimators" [2]. The key distinction from conventional supervised learning lies in the structure of each input (a dataset). Its "coordinates" are exchangeable, since the datapoints in each dataset are i.i.d., and different inputs can have different lengths (dataset sizes). Both properties affect the target learning rates, with our result showing how it decomposes into a pretraining term and a fresh-task term depending on the query context size NN.

Existing practical models perform in-context threshold inference

In-context threshold inference is practically relevant. MetaTree [4] is a direct example of a transformer trained to select tree splits in-context. Furthermore, state-of-the-art tabular foundation models like TabICL [3] and TabPFN [1] are trained with tree function priors and need to be able to infer thresholds in-context as a subroutine. We have ran new experiments showing that they do so close to optimally, nearly approaching the Bayes benchmark (Table R8), TabICL at every tested context size and TabPFN up to N=1024N=1024. At N=256N=256, their errors are 0.006140.00614 and 0.006480.00648, respectively, compared with 0.006040.00604 for the Bayes estimator. Our analysis, albeit in a simplified setting, show how such optimal behavior may be possible, and how it emerges from gradient descent training.

Table R8. Error on our task without fine-tuning. TabPFN and TabICL estimate the threshold as ∫01Pr⁡^(Y=1∣x) dx\int_0^1\widehat{\Pr}(Y=1\mid x)\,dx. For our one-head estimator, we report the best saved checkpoint for each NN (the final checkpoint gives 0.1390.139 at N=32N=32).

MethodN=32N=32N=64N=64N=256N=256N=1024N=1024N=4096N=4096
Bayes estimator BNB_N0.04390.04390.02310.02310.006040.006040.001570.001570.000400.00040
TabICL v20.04500.04500.02320.02320.006140.006140.001610.001610.000370.00037
TabPFN v20.05050.05050.02660.02660.006480.006480.002000.002000.001100.00110
Hard-attention limit MN+M_N^+0.09380.09380.04790.04790.01290.01290.003470.003470.000810.00081
One-head attention (ours)0.09500.09500.04810.04810.01560.01560.007890.007890.006340.00634

References:

[1] N. Hollmann et al. Accurate Predictions on Small Data with a Tabular Foundation Model. Nature, 637:319–326, 2025.

[2] M. Peyrard and K. Cho. Meta-Statistical Learning: Supervised Learning of Statistical Estimators. arXiv:2502.12088, 2025.

[3] J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. ICML, 2025.

[4] Y. Zhuang, L. Liu, C. Singh, J. Shang, and J. Gao. Learning a Decision Tree Algorithm with Transformers. arXiv:2402.03774, 2024.

AuthorsOpenReview ↗

Official Comment by Authors

Part 2/2

Our scalar query attention is closely connected to practical architecture

Our scalar head isolates a standard attention operation. A readout query attends to a set of observations, assigning each a dot-product score μXi+λYi\mu X_i+\lambda Y_i and pooling the feature values XiX_i. The column encoder in TabICL [3, Section 3.2] uses the same query-to-set aggregation operation across rows. The two-parameter restriction lets us track how gradient descent changes the balance between label and feature scores and how tightly the head concentrates.

What theoretical insights connect to practical models?

Our analysis reveals that in a noiseless threshold estimation problem, an attention head can learn to concentrate on the closest observation for a relevant class (positive class (Yi=1Y_i=1) vs. negative class (Yi=0Y_i=0)). This attention concentration is also observed in TabPFN and TabICL, with their most focused attention heads placing 44% and 82% of their attention mass, respectively, on the two observations closest to the threshold at just N=64N=64.

Furthermore, our analysis reveals that an attention head first learns to select the relevant label class, and then concentrates on the observations nearest the threshold within that class. To test whether this persists beyond our restricted model, we trained standard one- and four-layer transformers, with fully parameterized attention and MLP blocks, end to end from scratch on in-context threshold inference (one seed each). Specific heads pass through both phases in this order (Table R9). For example, head 0 of the one-layer model first moves its attention onto the positive class while its attention stays spread out, and only then concentrates near α\alpha. The same holds when the model must also infer which of four features to split on, where a head ends up concentrating along the split feature only. On the context sizes used in training, the errors of the one- and four-layer models are at most 1.111.11 and 1.281.28 times that of the Bayes estimator.

Table R9. Transformers trained from scratch. Each entry gives the attention mass on the class the head selects / the attention-weighted distance to α\alpha, along the split feature for the four-feature model (N=256N=256; uniform attention gives distance 0.2840.284). Each layer has 4 heads; layers and heads are numbered from 0. Each head first selects a class, then concentrates.

Training step1 layer: head 0 (positives)4 layers: layer 1, head 2 (negatives)4 layers: layer 3, head 2 (positives)4 layers, 4 features: layer 3, head 2 (positives)
1,0240.440.44 / 0.2770.2770.730.73 / 0.2550.2550.520.52 / 0.2840.2840.590.59 / 0.2840.284
2,0480.720.72 / 0.2850.2850.990.99 / 0.0530.0530.820.82 / 0.2650.2650.840.84 / 0.2750.275
4,0960.870.87 / 0.2050.2051.001.00 / 0.0220.0221.001.00 / 0.0470.0471.001.00 / 0.2100.210
Final1.001.00 / 0.00430.00431.001.00 / 0.00650.00651.001.00 / 0.00670.00671.001.00 / 0.0240.024

We agree that our paper may not have made this connections sufficiently explicit. We will clarify these connections in the introduction, signpost how the training analysis leads to the statistical guarantee, and enlarge the figure labels and define the plotted quantities in self-contained captions.

References:

[3] J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. ICML, 2025.

I apologise to the authors for the delayed reply, I have been busy with personal work.

I appreciate the effort that the authors have put into improving the manuscript given reviewer feedback -- the presentation has considerably improved. However, I feel like the additional experiments presented in the rebuttal do not quite address my concerns about practical motivation. My understanding is that the rebuttal illustrates the following two things

  1. Models like TabPFN that predict Y given X are able to infer implicitly the decision boundary.
  2. More complete transformer models, trained to explicitly predict the threshold also demonstrate similar attention concentration behaviour as discussed theoretically for the restricted model in this paper.

To me there seems to be a distinct motivational gap. The practical behaviour of interest is one where a threshold is inferred implicitly from inputs and labels, (e.g. TabPFN) where the threshold is never provided as explicit supervision or produced by the model as an explicit output. On the other hand, the theoretical scenario focused on by the paper is one where the decision threshold is provided as the training target. Inferring a threshold from inputs and labels because it is necessary for minimising a classification loss seems quite different to predicting a threshold using direct supervision. To illustrate my point, one (unlikely) way to close the gap would be to demonstrate a mechanism by which a model like TabPFN learns to internally predict the decision threshold value explicitly and then mechanistically use that for the final classification prediction.

I will re-iterate that as I am not a theoretical researcher, I will defer to other reviewers' opinions on the contribution of the paper. However, given the above, I will keep my current score.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for the follow-up. We agree that our model is pretrained on the threshold directly, whereas TabPFN and TabICL are pretrained on query labels; at inference, both see only a labeled context. In our setting, however, both objectives require the same computation. The context informs α\alpha only through the largest positive and the smallest negative observations, so for any prior the posterior predictive P(Yq=1∣xq,D)P(Y_q=1\mid x_q,D) is determined by these two boundary observations. A model trained on query labels must therefore locate them, even without producing α\alpha as an output. Consistent with this, at a query between these observations, the most concentrated heads of TabPFN and TabICL place a large share of their attention on these two observations, far more than uniform attention would. Our theory analyzes how gradient-based pretraining learns this localization with attention. Analyzing pretraining under a query-label objective remains open, and we will expand our discussion of this limitation in the revision.

Reviewer_steCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The general idea of the paper is that gradient descent can train a simple attention model to find the most decisive observations, those near the unknown decision boundary. The paper links training dynamics with the finiteness of pretraining data as well as estimation accuracy. The main contribution is the error-bound scaling, which comes from two different terms. The first one arising from the pretraining corpus (m and n) while the second one is due to new observations (N). Another interesting contribution is the infinite scale limit of the threshold error (μ\mu).

Critical strengths

The critical strength of the paper is the link between optimisation techniques and statistical accuracy. The general flow of the paper helps understanding sequentially what happens in this simplified attention model. This tractable framework allows for a careful analysis of pretraining and learning dynamics which allows for sharp predictions in how pretraining the model helps the estimator on a fresh task.

Critical weaknesses

One of the main weakness of the paper is the organisation. Especially, the paper lacks self-contained narrative in the main text. The paper should emphasise the reasoning from parameter calibration to attention concentration. Each experiment should identify what part of the theory it checks and provide detailed implications. Also on the experimental level, the choices in the parameters should be made clear and justified. Note that even though these comments are not invalidating the results, it does not allow to grasp what the paper is about. Then, essential experimental comparisons as well as needed discussions about parameter choices are relegated to the appendices and not barely cited in the main text. This is yet another difficulty of parsing through the text. Finally, it is possible to improve the notations in the text, for instance the subscripts are not always introduced (rad,fr..) and even the continuous gradient-flow time t is not discussed.

Questions for authors

-Is it possible to motivate the various experimental choices done in the Figure 1 of the main text? That is, is it sufficient to consider only the case where n=mn=m? There is the Figure 3 but implications for the error bound prediction is not trivial.

-Is the μ3μ˙\mu^3 \dot{\mu} curve stabilising in Figure 2? It does not seem the be the case up to μ=40\mu=40 but it is claimed in the text ("stabilize at constant scale")

-How important is Assumption 2.1 in general? Is it only needed for the proof?

-Is it possible to obtain the error when n and m vary separately?

-How much do the results depend on when training is stopped? Would the same trends appear if every model were trained for the same number of steps, or if stopping were based only on the observed training data?

-In Figure 5, does the poorer performance at small, nonzero initialisation reflect a longer transient? Or can these trajectories fail to reach the sharpening regime?

8 comments
AuthorsOpenReview ↗

Official Comments by Authors

Part 1/2

We thank the reviewer for recognizing the link between optimization and statistical accuracy, and that our tractable framework yields sharp predictions of how pretraining helps on fresh tasks. We respond to the questions and weaknesses below.

Q1 and Q4: Varying the number and size of pretraining tasks

Q1. We set m=nm=n in Figure 1 to show how fresh-task error changes when both pretraining dimensions grow together. Appendix H.2 also states the errors for unequal configurations, obtained under the same checkpoint protocol, and Table R6 below reports a rerun of these configurations alongside our new stopping rule. Figure 3 measures gradient error at a fixed attention scale, while these comparisons measure the fresh-task error of the frozen heads. We will summarize the unbalanced comparisons in Section 6.

Q4. Theorem 3.1 applies to unequal mm and nn. At concentration μ≍R\mu\asymp R, Equation (14) bounds the normalized gradient error by C{RL/m+RL/m+R/n}C\{\sqrt{RL/m}+RL/m+R/n\} with probability at least 1−δ1-\delta, where L=log⁡(C(1+R)mn/δ)L=\log(C(1+R)mn/\delta) and n≥CRLn\ge CRL. Increasing mm reduces fluctuations across pretraining tasks, while increasing nn reduces finite-context bias. These controls certify resolution of order Am,n=(m∧n)/log⁡(C1mn)A_{m,n}=(m\wedge n)/\log(C_1mn) and give a fresh-task error bound of O~((m∧n)−1+N−1)\widetilde O((m\wedge n)^{-1}+N^{-1}). Up to logarithmic factors, increasing mm with nn fixed improves the certified pretraining term until mm reaches nn, and increasing nn with mm fixed gives the analogous dependence.

Q2: μ3μ˙\mu^3\dot\mu in Figure 2 remains bounded

Theorem 5.2 gives only 0<c≤μ3μ˙≤C0<c\le\mu^3\dot\mu\le C, and we do not claim convergence by μ=40\mu=40; we will replace "stabilize at constant scale" with "remain at constant order". Over longer training, μ3μ˙\mu^3\dot\mu stays between 0.4960.496 and 0.5290.529 for all 234 initializations of Figure 5.

Q5: Sensitivity to training duration and stopping from observed data

Same number of steps. In Figure 1, larger corpora were trained longer to reach a higher target concentration. We therefore trained all nine (m,n)∈{128,256,512}2(m,n)\in\{128,256,512\}^2 configurations for common budgets of up to 1,536,0001,536,000 GD iterations, using two corpora per configuration (Table R5). At matched steps, the errors are nearly identical across configurations at N=64N=64 and N=4096N=4096. This suggests that much of the corpus-size trend in Figure 1 at these context lengths comes from the different training budgets. At N=32N=32, the configurations diverge late in training, driven by the threshold α=0.25\alpha=0.25 near the lower end of the task range. At that threshold, a short context can lack a positive observation (Yi=1Y_i=1) close to the split, and sharper attention can then favor a negative observation (Yi=0Y_i=0). This deterioration occurs well after the Figure 1 checkpoints. We will describe the plateau in Figure 1 as checkpoint-dependent and summarize the matched-step results in Section 6.

Table R5. Error at matched GD iterations. Ranges are across the nine configurations (m,n)∈{128,256,512}2(m,n)\in\{128,256,512\}^2, each summarized by the median over two corpora. All runs use the same initialization and γ=400\gamma=400.

GD iterationsN=32N=32N=64N=64N=4096N=4096
470.1550.155–0.1630.1630.1230.123–0.1260.1260.0860.086–0.0880.088
1,5000.1050.105–0.1060.1060.0670.067–0.0680.0680.0330.033–0.0330.033
24,0000.0940.094–0.1100.1100.0530.053–0.0530.0530.01620.0162–0.01660.0166
384,0000.0910.091–0.2860.2860.0480.048–0.0480.0480.00830.0083–0.00850.0085
1,536,0000.1110.111–0.4980.4980.0470.047–0.0470.0470.00600.0060–0.00620.0062

Stopping from observed training data. We kept Figure 1's concentration target Rm,nR_{m,n}, the largest value on the checkpoint grid at or below (m∧n)/log⁡(mn)(m\wedge n)/\log(mn), and stopped when the empirical head's learned concentration μ^sγ\widehat\mu_s^\gamma first reached it. This rule uses the fitted parameters and removes the need for a population reference trajectory. It gives similar stopping times and fresh-task errors at N=4096N=4096 (Table R6).

Table R6. Population-reference versus observed-parameter stopping at the same target Rm,nR_{m,n}. Observed iterations and errors are medians over 4–5 corpora per configuration.

(m,n)(m,n)Rm,nR_{m,n}Number of GD iterations PopulationNumber of GD iterations ObservedError at N=4096N=4096 PopulationError at N=4096N=4096 Observed
(64,64)(64,64)6560.16820.16820.16060.1606
(128,128)(128,128)1247470.08620.08620.08680.0868
(128,512)(128,512)81214.50.13270.13270.12370.1237
(512,128)(512,128)81212.50.12670.12670.12540.1254
(256,256)(256,256)181801730.05810.05810.05880.0588
(256,512)(256,512)18180183.50.05910.05910.05880.0588
(512,256)(512,256)18180178.50.05880.05880.05880.0588
(512,512)(512,512)40325231450.02710.02710.02740.0274
AuthorsOpenReview ↗

Official Comments by Authors

Part 2/2

Q3 and Q6: Small initialization only lengthens the initial transient

Assumption 2.1 (large μ0\mu_0 initialization) is used in our proof to ensure our asymptotic approximations are valid. Practically, it is clear that some assumption is needed for convergence as μ0=0\mu_0 = 0 is stationary by symmetry. To test what happens with smaller μ0\mu_0, we extended the 234 initializations of Figure 5 to flow time T=106T=10^6 (Table R7). Smaller starts stay longer on the initial plateau, but by T=106T=10^6 all reach risk between 2.5×10−42.5\times10^{-4} and 3.3×10−43.3\times10^{-4}, and every start then shows the calibration and μ4\mu^4 growth of Theorem 5.2. Minibatch SGD from random initialization behaves the same way in all 80 runs (Table R4 in our response to Reviewer duKR).

Table R7. Population risk at flow time TT, range over the 234 initializations of Figure 5 (all 13 ratios k0k_0 per row).

Initial concentration μ0\mu_0T=103T=10^3T=104T=10^4T=105T=10^5T=106T=10^6
[10−3,10−2)[10^{-3},10^{-2})1.5×10−21.5\times10^{-2}1.4×10−21.4\times10^{-2}–1.5×10−21.5\times10^{-2}1.2×10−31.2\times10^{-3}3.3×10−43.3\times10^{-4}
[10−2,1)[10^{-2},1)1.4×10−21.4\times10^{-2}–1.5×10−21.5\times10^{-2}4.8×10−34.8\times10^{-3}–1.4×10−21.4\times10^{-2}1.1×10−31.1\times10^{-3}–1.2×10−31.2\times10^{-3}3.3×10−43.3\times10^{-4}
[1,30][1,30]5.3×10−45.3\times10^{-4}–1.2×10−11.2\times10^{-1}5.0×10−45.0\times10^{-4}–4.8×10−34.8\times10^{-3}4.0×10−44.0\times10^{-4}–1.1×10−31.1\times10^{-3}2.5×10−42.5\times10^{-4}–3.3×10−43.3\times10^{-4}
All 234 starts5.3×10−45.3\times10^{-4}–1.2×10−11.2\times10^{-1}5.0×10−45.0\times10^{-4}–1.5×10−21.5\times10^{-2}4.0×10−44.0\times10^{-4}–1.2×10−31.2\times10^{-3}2.5×10−42.5\times10^{-4}–3.3×10−43.3\times10^{-4}

Organization, experimental choices, and notation We agree and will revise accordingly. We will organize the main text around the chain from calibration to concentration to accuracy, state which prediction each experiment tests and justify its parameter choices, and add a notation paragraph that defines all subscripts.

Reviewer_steCOpenReview ↗

Acknowledgment of the small initialisation check

Thank you for the reply. The Table R7 helps realising the role a smaller initialisation plays in the simulation. However, it raises additional questions. In particular: is it possible to discuss the order of magnitude of a large initialisation from theory? Because in the paper and in the reply, large enough is never characterised precisely. Also, the Figure 5 suggests a non trivial role of this value of μ0\mu_0 which is not discussed either (to the best of my knowledge). It is suggested in the Assumption 2.1 that it should scale with ρ\rho and kk, this is not discussed either in the paper. A more precise discussion on the role the initialisation plays would also help the clarity of the paper.

Reviewer_duKROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies a tractable model of in-context learning, where a one-head softmax attention is trained with gradient descent to estimate a decision threshold from labeled examples. The authors show that the trained model estimates the threshold of a new task with an error that splits into a part due to the finite number and size of training tasks, and a part depending on the size of the new context, approaching the rate of the optimal statistical estimator. Their analysis also describes the training dynamics of the attention parameters, and is supported by numerical experiments.

Critical strengths

The paper provides a novel, sound analysis, giving interpretable and quantitative insights on the sources of error and on the training dynamics of the attention parameters. The authors are transparent about assumptions and limitations, and the experiments are consistent with the theory.

Critical weaknesses
  1. The readability and clarity of the paper are affected by several issues that make the text harder to parse. Notation is often introduced before being explained, or not explained in the main text (e.g. the attention parameters in l. 129 are introduced without explanation; the interval Im,n,γI_{m,n,\gamma} and the notation θ^s^γ\hat\theta^{\gamma}_{\hat s} in Theorem 3.1 are only clarified later). Several paragraphs meant to give intuition (e.g. l. 225–230, 276–278, 308–313), as well as the experimental section, are very compressed and rely on terminology that is not clear from the main text (e.g. "one-step barrier argument", "logarithmic tube", "axis-swapped configurations"), which may be unfamiliar to a broad audience.
  2. While the authors motivate the choice of their setting and are open about its limitations, the paper does not sufficiently discuss what its findings imply beyond it, for instance by relating them to empirical observations or providing insights on more realistic models.
Questions for authors
  1. Could the authors clarify the relevance of their results beyond the considered setting?
  2. Would a small initialization only delay the dynamics, or could it change the paper's conclusions?
4 comments
AuthorsOpenReview ↗

Official Comment by Authors

Part 1/2

We thank the reviewer for finding our analysis novel and sound, with interpretable and quantitative insights into the sources of error and the training dynamics, and for noting our transparency about assumptions and limitations. We respond to the questions and presentation points below.

Q1: Our analysis is relevant beyond the considered setting

Existing practical models perform in-context threshold inference. MetaTree [3] is a direct example of a transformer trained to select tree splits in-context. Furthermore, state-of-the-art tabular foundation models like TabICL [2] and TabPFN [1] are trained with tree function priors and need to be able to infer thresholds in-context as a subroutine. We have ran new experiments showing that they do so close to optimally, nearly approaching the Bayes benchmark (Table R1), TabICL at every tested context size and TabPFN up to N=1024N=1024. At N=256N=256, their errors are 0.006140.00614 and 0.006480.00648, respectively, compared with 0.006040.00604 for the Bayes estimator. Our analysis, albeit in a simplified setting, show how such optimal behavior may be possible, and how it emerges from gradient descent training.

Table R1. Error on our task without fine-tuning. TabPFN and TabICL estimate the threshold as ∫01Pr⁡^(Y=1∣x) dx\int_0^1\widehat{\Pr}(Y=1\mid x)\,dx. For our one-head estimator, we report the best saved checkpoint for each NN (the final checkpoint gives 0.1390.139 at N=32N=32).

MethodN=32N=32N=64N=64N=256N=256N=1024N=1024N=4096N=4096
Bayes estimator BNB_N0.04390.04390.02310.02310.006040.006040.001570.001570.000400.00040
TabICL v20.04500.04500.02320.02320.006140.006140.001610.001610.000370.00037
TabPFN v20.05050.05050.02660.02660.006480.006480.002000.002000.001100.00110
Hard-attention limit MN+M_N^+0.09380.09380.04790.04790.01290.01290.003470.003470.000810.00081
One-head attention (ours)0.09500.09500.04810.04810.01560.01560.007890.007890.006340.00634

Our scalar query attention is closely connected to practical architecture. Our scalar head isolates a standard attention operation. A readout query attends to a set of observations, assigning each a dot-product score μXi+λYi\mu X_i+\lambda Y_i and pooling the feature values XiX_i. The column encoder in TabICL [2, Section 3.2] uses the same query-to-set aggregation operation across rows. The two-parameter restriction lets us track how gradient descent changes the balance between label and feature scores and how tightly the head concentrates.

Two insights which carry over to more standard models.

First, our analysis reveals that in a noiseless threshold estimation problem, an attention head can learn to concentrate on the closest observation for a relevant class (positive class (Yi=1Y_i=1) vs. negative class (Yi=0Y_i=0)). This attention concentration is also observed in TabPFN and TabICL, with their most focused attention heads placing 44% and 82% of their attention mass, respectively, on the two observations closest to the threshold at just N=64N=64.

Second, our analysis reveals that training reaches this in two phases. An attention head first learns to select the relevant label class, and only then concentrates on the observations nearest the threshold within that class. To test whether this persists beyond our restricted model, we trained standard one- and four-layer transformers, with fully parameterized attention and MLP blocks, end to end from scratch on in-context threshold inference (one seed each). Specific heads pass through both phases in this order (Table R2). For example, head 0 of the one-layer model first moves its attention onto the positive class while its attention stays spread out, and only then concentrates near α\alpha. The same holds when the model must also infer which of four features to split on, where a head ends up concentrating along the split feature only. On the context sizes used in training, the errors of the one- and four-layer models are at most 1.111.11 and 1.281.28 times that of the Bayes estimator.

References:

[1] N. Hollmann et al. Accurate Predictions on Small Data with a Tabular Foundation Model. Nature, 637:319–326, 2025.

[2] J. Qu, D. Holzmüller, G. Varoquaux, and M. Le Morvan. TabICL: A Tabular Foundation Model for In-Context Learning on Large Data. ICML, 2025.

[3] Y. Zhuang, L. Liu, C. Singh, J. Shang, and J. Gao. Learning a Decision Tree Algorithm with Transformers. arXiv:2402.03774, 2024.

AuthorsOpenReview ↗

Official Comment by Authors

Part 2/2

Q1: Our analysis is relevant beyond the considered setting (continued)

Table R2. Transformers trained from scratch. Each entry gives the attention mass on the class the head selects / the attention-weighted distance to α\alpha, along the split feature for the four-feature model (N=256N=256; uniform attention gives distance 0.2840.284). Each layer has 4 heads; layers and heads are numbered from 0. Each head first selects a class, then concentrates.

Training step1 layer: head 0 (positives)4 layers: layer 1, head 2 (negatives)4 layers: layer 3, head 2 (positives)4 layers, 4 features: layer 3, head 2 (positives)
1,0240.440.44 / 0.2770.2770.730.73 / 0.2550.2550.520.52 / 0.2840.2840.590.59 / 0.2840.284
2,0480.720.72 / 0.2850.2850.990.99 / 0.0530.0530.820.82 / 0.2650.2650.840.84 / 0.2750.275
4,0960.870.87 / 0.2050.2051.001.00 / 0.0220.0221.001.00 / 0.0470.0471.001.00 / 0.2100.210
Final1.001.00 / 0.00430.00431.001.00 / 0.00650.00651.001.00 / 0.00670.00671.001.00 / 0.0240.024

The learned estimator remains accurate under non-uniform features. Under five non-uniform feature densities, the model trained under uniform features keeps nearly the same error at N=4096N=4096; at N=256N=256, densities that are small near an edge of the task range raise the error by up to about 2×2\times.

Q2: Small initialization only delays the dynamics

Assumption 2.1 is used only to make our asymptotic estimates valid from the start of training. Exactly zero initialization is stationary by symmetry, but every small positive start eventually follows the same path. We extended the 234 initializations of Figure 5, down to μ0=10−3\mu_0=10^{-3}, to flow time T=106T=10^6 (Table R3). Smaller starts stay longer on the initial plateau, but by T=106T=10^6 all reach risk between 2.5×10−42.5\times10^{-4} and 3.3×10−43.3\times10^{-4}, and every start then shows the calibration and μ4\mu^4 growth of Theorem 5.2, with μ3μ˙\mu^3\dot\mu between 0.4960.496 and 0.5290.529.

Table R3. Population risk at flow time TT, range over the 234 initializations of Figure 5 (all 13 ratios k0k_0 per row).

Initial concentration μ0\mu_0T=103T=10^3T=104T=10^4T=105T=10^5T=106T=10^6
[10−3,10−2)[10^{-3},10^{-2})1.5×10−21.5\times10^{-2}1.4×10−21.4\times10^{-2}–1.5×10−21.5\times10^{-2}1.2×10−31.2\times10^{-3}3.3×10−43.3\times10^{-4}
[10−2,1)[10^{-2},1)1.4×10−21.4\times10^{-2}–1.5×10−21.5\times10^{-2}4.8×10−34.8\times10^{-3}–1.4×10−21.4\times10^{-2}1.1×10−31.1\times10^{-3}–1.2×10−31.2\times10^{-3}3.3×10−43.3\times10^{-4}
[1,30][1,30]5.3×10−45.3\times10^{-4}–1.2×10−11.2\times10^{-1}5.0×10−45.0\times10^{-4}–4.8×10−34.8\times10^{-3}4.0×10−44.0\times10^{-4}–1.1×10−31.1\times10^{-3}2.5×10−42.5\times10^{-4}–3.3×10−43.3\times10^{-4}
All 234 starts5.3×10−45.3\times10^{-4}–1.2×10−11.2\times10^{-1}5.0×10−45.0\times10^{-4}–1.5×10−21.5\times10^{-2}4.0×10−44.0\times10^{-4}–1.2×10−31.2\times10^{-3}2.5×10−42.5\times10^{-4}–3.3×10−43.3\times10^{-4}

We also ran minibatch SGD from random initialization, (λ0,μ0)∼N(0,σ2I)(\lambda_0,\mu_0)\sim\mathcal N(\mathbf 0,\sigma^2\mathbf I) with σ∈{1,0.01}\sigma\in\{1,0.01\} (Table R4). All 80 runs leave the origin within six steps, calibrate and sharpen as predicted, and reach the error of full-batch GD from our initialization at matched concentration. Runs that start with μ<0\mu<0 learn the mirror-image head for the reflected task.

Table R4. Minibatch SGD from random initialization. Each entry gives the median ∣μ∣|\mu| / error at N=64N=64 over 40 runs per σ\sigma, compared with full-batch GD from our initialization at similar μ\mu (two corpora in the last row).

SGD stepσ=1\sigma=1: $\vert \mu\vert $ / errorσ=0.01\sigma=0.01: $\vert \mu\vert $ / errorFull-batch GD: μ\mu / error
11.91.9 / 0.3570.3570.80.8 / 0.3490.349–
6413.213.2 / 0.1160.11613.113.1 / 0.1170.11712.012.0 / 0.1250.125
1,02429.529.5 / 0.0750.07529.529.5 / 0.0750.07532.532.5 / 0.0710.071
131,072105.6105.6 / 0.05520.0552106.0106.0 / 0.05520.055297.397.3 / 0.05570.0557; 116.3116.3 / 0.05460.0546

We agree that our paper may not have explained its notation and terminology sufficiently. We will explain all notation where it is introduced, and replace specialized terms in the intuition paragraphs and the experimental section by their meaning.

I thank the authors for their detailed response and the revisions to the manuscript, which improve clarity, although several passages remain dense, and address my questions. I raise my score to weak accept.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you very much for the follow-up and for raising your score. We are glad the revisions addressed your questions, and we will further improve the readability of the remaining dense passages in the revision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We thank the reviewers for their careful reading. They found the analysis novel and sound, with interpretable insights into the sources of error and the training dynamics (duKR), valued the link it draws between optimization and statistical accuracy (steC), and noted the substantial technical work and the correctness of the theory (wj3P). Most questions concerned how our theoretical result relates to practice. We ran additional experiments to answer them, with full results in our individual responses, and all of them are consistent with the theory.

  • duKR (practical models, initialization). Pretrained tabular foundation models perform our task close to the Bayes estimator, and small or random initializations only delay the dynamics.
  • steC (roles of mm, nn, stopping time, initialization). At matched iterations, all nine (m,n)(m,n) give nearly identical errors, so the corpus-size trend in Figure 1 largely reflects training duration.
  • wj3P (relevance to practical in-context learning). Transformers trained from scratch pass through the predicted calibration and sharpening phases.

Manuscript updates (in blue; global notation changes, namely bold vectors and matrices and symbols replacing text subscripts, are not marked)

  • Abstract and Introduction. State when the rate is Bayes-optimal, motivate the problem through in-context learning of decision trees, and signpost how the training analysis leads to the accuracy guarantee.
  • Sections 2–6 (duKR-W1; steC-W; wj3P-W2). Add a notation paragraph and the connection to TabICL, define all quantities before Theorem 3.1, and state the experimental choices and the prediction each experiment tests.
  • Section 8. Discusses implications for pretrained and end-to-end trained transformers.
  • Appendices A and H. Add the Bayes and one-head risk constants, complete the experimental settings, and present the new experiments (H.4–H.6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.