- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates whether transformers can act as in-context samplers, generating new samples from an unknown distribution based on examples provided in the input context, without updating model parameters. The authors theoretically demonstrate that transformers can simulate closed-form diffusion samplers and approximate an estimation-free sampling (EFS) procedure through specifically constructed attention and feedforward operations. The paper presents synthetic point-cloud experiments to demonstrate in-context sampling and analyzes the layerwise representations of pretrained language models.
Critical strengths
-
The paper explores a theoretical connection between transformers and generative sampling algorithms. Extending in-context learning from conventional prediction tasks to distribution sampling is a potentially meaningful research direction.
-
The theoretical constructions are technically sound, particularly the interpretation of softmax attention as computing Gaussian responsibility weights and feedforward layers as implementing iterative sampling updates.
Critical weaknesses
-
The experiments do not convincingly demonstrate the central claimed capability of in-context sampling. The primary empirical evidence is the smile experiment in Figure 1, where a transformer trained without the smile component generates points along a smile-shaped curve provided through in-context samples. However, I do not think this experiment is sufficient to establish meaningful generalization to unseen distributions. First, the smile-shaped curve is geometrically similar to the circular face boundary included in the training distribution. Both consist of smooth circular arcs. Therefore, generating points along the smile does not necessarily require learning a new distributional structure. The model may simply transfer a familiar local geometric pattern to a different spatial location. Second, and more importantly, the in-context samples in Figure 1(d) already appear to cover the entire region occupied by the predicted tokens. The generated points are located within the geometry defined by the provided samples, rather than demonstrating the ability to infer substantially new structures. Simple interpolation, nearest-neighbor resampling, or kernel density estimation (KDE) could also explain this behavior. None of these explanations requires the transformer to learn a sophisticated or generalizable sampling algorithm. Third, the paper does not compare the transformer with these simple baselines. Without such comparisons, it is difficult to determine whether the transformer demonstrates any meaningful capability beyond local interpolation or empirical-distribution resampling. More convincing experiments would involve training and testing on distributions with substantially different geometric structures, such as training on circular distributions and testing on triangular or multimodal distributions. Importantly, the evaluation should examine whether the generated samples reproduce the target probability distribution, rather than merely following the geometric outline of the in-context points. Overall, Figure 1 provides evidence that the model's outputs depend on the in-context samples, but this is substantially weaker than demonstrating generalizable in-context sampling. Since this capability is central to the paper's motivation, I consider the lack of convincing empirical validation a major weakness.
-
The theoretical results do not explain the mechanisms learned by trained transformers. The theoretical contributions show that specifically constructed transformer parameters can implement closed-form diffusion and EFS-inspired sampling algorithms. However, demonstrating that a transformer architecture can represent a particular algorithm is different from showing that a transformer trained on data actually learns that algorithm. The paper does not connect the constructed transformer parameters to those obtained through the training procedure. In particular, there is no direct evidence that the trained transformers perform the iterative sampling updates described in the theoretical sections.
-
The evaluation lacks quantitative evidence of distribution-level generalization. In-context sampling should be evaluated by whether the generated samples follow the probability distribution specified by the context, rather than simply whether they appear geometrically plausible. For example, two distributions may share exactly the same triangular support but assign different probability masses to different edges. A model that generates points along the triangle boundary may appear successful visually, even if it fails to reproduce the correct probability density. The paper does not systematically evaluate such cases. Quantitative metrics such as MMD or Wasserstein distance, together with experiments varying context size, distribution geometry, and probability density, would be necessary to establish the extent of the model's sampling capability.
Questions for authors
-
How does the smile experiment establish genuine in-context sampling rather than interpolation? The smile geometry resembles the circular boundary in the training distribution, and the predicted samples in Figure 1(d) appear to lie within the region already covered by the in-context samples. Could the authors compare against interpolation, nearest-neighbor resampling, and KDE baselines? Could the authors also evaluate test distributions with substantially different geometries from those encountered during training?
-
Can the authors show that the generated samples reproduce the target probability distribution, not just its geometric support? For example, can the model distinguish distributions with identical geometric support but different probability densities? Quantitative comparisons using MMD or Wasserstein distance, along with evaluations across different context sizes, would help establish whether the model learns generalizable distribution-sampling behavior.
-
What evidence connects the theoretical sampling constructions to the actual computations performed by trained transformers? The theorems establish that transformer parameters can implement diffusion or EFS-inspired procedures, but they do not show that training produces these mechanisms. Can the authors provide direct comparisons between learned layerwise updates and the updates prescribed by the proposed sampling algorithms?
3 comments
I thank the authors for their response, which addresses some of my concerns, but the main issues are still unresolved. Specifically, the transformer still performs worse than simple baselines, which makes me concerned about its ability to learn a generalizable sampling algorithm. I will maintain my original ratings.
We agree that the baselines achieve lower distributional error. This is what we expect; if it wasn't like that, we would be surprised. For copying and resampling, there is a mathematical reason to expect a small MMD. Let be independent draws from , and let be a uniformly chosen index independent of the prompt. For any measurable set ,
Thus both uniform resampling and copying the last point have output law across prompts. MMD compares the mean kernel features of two distributions, so their population discrepancy is . Even a sampler producing fresh, exact draws from can only tie that score.
For the face experiments, we compare outputs from independent prompts with independent target points. For either copying rule, the expected empirical squared MMD is
where are independent draws from . This uses the Gaussian kernel and the estimator with diagonal terms in Appendix F.3. In the scalar experiments, the reference is drawn from the pooled contexts. Uniform resampling also reproduces that pool in expectation. These observations explain why copying and resampling are strong baselines.
Fu et al. (came on arxiv after the ICLR deadline) study a related setting where the model must generate an output for a new query. Copying a demonstration output need not give the correct distribution for that query. Their theorem gives a total-variation guarantee for a transformer pretrained across tasks. Fu et al., Diffusion Transformers are Provably Optimal In-context Generators
We added triangular-mouth tests using a transformer trained on circular outlines and eyes, without mouths; Figures F.1 and F.2 show the results.
We have compared the trained models with copying the last point, context resampling, local interpolation, and KDE in Appendix F of the revised submission draft. In these experiments, the transformer’s sampling patterns and MMD values differ from those of copying the last point, random context resampling, KDE, and local interpolation. We therefore do not believe that the transformer output can be explained solely by copying or direct resampling of context points.
When the U-shaped profile holds for the energy function, we can prove that the implemented algorithm is approximately EFS. This conclusion follows from the gradient theorem. Let
Δℓ=Xℓ+1−Xℓ,γ(s)=Xℓ+sΔℓ,0≤s≤1.
Assuming E is continuously differentiable, we have
E(Xℓ+1)−E(Xℓ)=∫01⟨∇E(γ(s)),γ′(s)⟩F,ds=⟨Gℓ,Δℓ⟩F.
Gℓ=∫01∇E(Xℓ+sΔℓ),ds.
Here Gℓ is the gradient averaged along the segment between consecutive layer representations. When energy decreases, the update has a negative projection onto this averaged gradient. When energy increases, the projection is positive. Consequently, a U-shaped sequence of energies implies
A negative projection onto the averaged gradient of energy during the descending phase. If energy decreases by at least aℓ>0 in layer ℓ, then ⟨−Gℓ,Δℓ⟩F≥aℓ.
A positive projection onto the averaged gradient of energy during the ascending phase. If energy increases by at least bℓ>0 in layer ℓ, then ⟨Gℓ,Δℓ⟩F≥bℓ.
This gives a precise relation between the U-shaped energy profile and EFS. The transformer’s energy first decreases and then increases across layers, consistent with the gradient descent and ascent phases of EFS.
We will therefore clarify the scope of our claims. The constructions prove that transformers can implement the proposed EFS sampling algorithms. The U-shaped profiles provide a reason to investigate EFS as a method closely resembling the learned algorithm.
We tested the same triangle with edge probabilities (1/3,1/3,1/3) and (0.7,0.2,0.1). For the second case, the transformer produces nearest-edge masses of approximately (0.578,0.295,0.127), showing sensitivity to density. Figure F.4 and Table F.6 report results for context sizes 16, 32, and 64.