acceptodds
Under review as a conference paper at ICLR 2027

Diffusion Transformers are Provably Optimal In-context Generators

Abstract

Generative foundation models are attracting interest for their ability to produce desired outputs from demonstrations given at inference time, without updating parameters. However, since a few demonstrations cannot uniquely identify the intended task, the challenge is how to learn and sample from an output distribution that reflects this task uncertainty. In this work, we theoretically analyze how a Diffusion Transformer (DiT), pretrained across diverse tasks, learns and generates predictive distributions for a new query from demonstrations. We first show that the natural target to generate from finite demonstrations is not an output derived from estimating a single task, but rather a predictive distribution that captures the task uncertainty remaining after observing the demonstrations. We then prove that a DiT can learn this predictive distribution through score estimation, using attention to aggregate information from demonstrations and diffusion to generate samples. Owing to this property, with sufficient pretraining resources and diffusion sampling steps, the resulting DiT achieves the minimax optimal rate over a Hölder class of test-time tasks. These results imply that DiT acts as a statistically grounded in-context generator capable of generating distributions adapted to new tasks while retaining the uncertainty inherent in finite demonstrations.

Primary area:
learning theory
Keywords:
In-context generation, Diffusion Transformer, Minimax optimal rates

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_thgpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a theoretical paper on in-context generation with Diffusion Transformers (DiT). The authors argue that the right target is to generate samples from the posterior-predictive distribution rather than a single output. They analyze a simplified prompt-conditioned DiT and show that it can learn and sample from this distribution from pretraining episodes, without task labels, using polynomial model size, data, and sampling steps. They further prove that the error due to a finite prompt scales in terms of the number of demonstrations n, with a matching lower bound, so the trained DiT attains the minimax-optimal rate in prompt length. A small-scale synthetic image experiment illustrates the predicted behavior (bimodal predictive distribution under short contexts, contracting as context grows). Overall this is a thorough theoretical contribution with a clear conceptual message.

Critical strengths

• Clear formulation of in-context generation as sampling from the posterior-predictive distribution. The main message — that in-context inference is Bayesian posterior prediction and uncertainty should be preserved — is already present in prior work the paper cites; the contribution is extending it to generation and proving it rigorously. • Comprehensive theoretical derivation covering the full pipeline from model capacity through learning from data to sampling, with a matching lower bound

Critical weaknesses

• The analyzed model differs substantially from DiTs used in practice. The experiment uses a conventional DiT rather than the analyzed one, so the connection between the theorems and practical DiTs is asserted rather than demonstrated. • The experiment shows the qualitative behavior (bimodality, contraction with more demonstrations) but do not directly evaluate the theoretical error rates for a learned DiT.

Questions for authors

• Does equation (10) give practical guidance? Appendix G.9 gives N = Õ(n⁶) and M = Õ(n) for the scalar case, while the experiment used about 10⁶ episodes at n ≤ 64. Can the authors reconcile this gap? • The empirical evaluation is limited to controlled synthetic settings. While these examples illustrate the theoretical results, experiments on more realistic in-context generation tasks would strengthen the practical relevance of the work.

5 comments
AuthorsOpenReview ↗

Response to Reviewer thgp (1/2)

Thank you for the positive assessment and the suggestions. We address your concerns and have added experiments on Fashion-MNIST (Appendix J.3).

W1. Gap between the analyzed model and practical DiTs.

We agree that our theorems apply to a simplified DiT rather than to conventional DiTs, and we have clarified this difference in the revised manuscript (Remark E.1). Our theory still connects to practical DiTs in three ways.

  • Architecture-independent results. The error decomposition and the finite-prompt bounds hold for any generator, including practical DiTs, and the learning and sampling analyses apply to any score class with suitable approximation and complexity bounds. Our experiments measure the quantities in these results: the posterior-predictive error and the finite-prompt term.

  • Relevance to practical DiT experiment. Our experiments support the main qualitative predictions of the theory. While our theory focuses on the stylized architecture, we empirically observe similar qualitative trends in a standard DiT, suggesting that these phenomena are not specific to the simplified model.

  • Extension to practical DiT's theory. While our work represents a starting point, we believe this theoretical framework has the potential to be extended to more standard DiT architectures. We have added two possible directions in Remark E.2: extending our explicit construction to standard DiTs or directly analyzing their approximation capabilities. The main challenge is to establish polynomial approximation and complexity bounds while accounting for layer normalization, patch tokenization, and learned MLPs.

W2. Do the experiments evaluate the theoretical error rates?

We now measure category-level generation error as pretraining and prompt length vary, and compare it with the exact finite-prompt benchmark, which is known at the category level (Fashion-MNIST, 24 test tasks). These experiments examine a fixed family, while the minimax rate concerns the worst case over the full class.

  • Pretraining (n=64n=64, Figure 7, left; 13 checkpoints): from 0.0320.032 to 15.0715.07 million episodes, the DiT's TV to the realized mixture falls from 0.5140.514 to 0.1040.104, and its TV to the posterior predictive falls from 0.5090.509 to 0.0560.056.
  • Context length (final checkpoint, Figure 7, right): from n=4n=4 to 6464, the DiT's TV to the realized mixture falls from 0.2190.219 to 0.1000.100, following the same trend as the posterior predictive (green curve).

Each image belongs to one category and is drawn independently of the task given its category, so the label-based reference equals the image-based one at the category level. For this category mixture, an argument analogous to Proposition I.1 gives a finite-prompt term of order n−1/2n^{-1/2} without a logarithm. We used 100 DDIM steps and did not vary the number of sampling steps.

AuthorsOpenReview ↗

Response to Reviewer thgp (2/2)

Q1. Does Eq. (10) give practical guidance, and why are the experimental budgets much smaller?

It gives qualitative guidance. Eq. (9) bounds the posterior-predictive error by ε\varepsilon plus an estimation term O~(Nparam,n,εlat/N)\widetilde{O}(\sqrt{\mathsf{N}_{\mathrm{param},n,\varepsilon}^{\mathrm{lat}}/N}) and a sampling term O~(M−1/2)\widetilde{O}(M^{-1/2}). Here Nparam,n,εlat\mathsf{N}_{\mathrm{param},n,\varepsilon}^{\mathrm{lat}} is the number of trainable parameters, NN the number of pretraining episodes, and MM the number of sampling steps. Eq. (10) gives the resulting sufficient NN and MM for error O(ε)O(\varepsilon). More pretraining or sampling does not reduce the separate finite-prompt term. Appendix K turns this into diagnostics: with the checkpoint fixed, increase the sampler steps. If the output still differs from the posterior predictive, improve training, capacity, or use of context. Once these errors are small, more informative demonstrations help most.

The experimental budgets are far below N=O~(n6)N=\widetilde{O}(n^6) because that bound covers the worst case over the Hölder class, for which the scalar construction uses O~(n)\widetilde{O}(n) representative tasks. Our rendered-shape family is much simpler: its posterior depends on the context only through five sums, and its predictive distribution is a two-component Gaussian mixture. Parameter-count bounds are also loose for overparameterized networks, and the step bound concerns the randomized SDE sampler rather than DDIM.

Q2. More realistic in-context generation tasks.

The Fashion-MNIST experiments move from rendered shapes to real images while keeping the category-level posterior predictive known, so that calibration can be checked directly.

  • Task. Each episode draws category proportions from Dirichlet(0.5,…,0.5)\mathrm{Dirichlet}(0.5,\ldots,0.5) and samples context and target images from that mixture.

  • Result. When we vary the category composition of a 64-image context (Figure 8, all ten category pairs), the generated frequency of the second category increases at every tested step. Although the task structure is still controlled, we confirmed that the results are consistent with our theory.

AuthorsOpenReview ↗

Gentle Reminder

Thank you again for your time and helpful feedback. As the discussion period is coming to a close, we wanted to follow up. We have expanded the discussion of how our construction relates to practical DiTs and added Fashion-MNIST experiments that measure generation error as pretraining and context length vary. We would appreciate your thoughts and would be happy to clarify any remaining concerns.

Best regards,

The Authors

Reviewer_GiBLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates in-context generation. First, the authors argue that, because demonstrations may not uniquely identify the underlying task, the model should generate from a predictive distribution that accounts for multiple plausible tasks rather than selecting a single task and generating from its distribution. The paper provides a theoretical analysis using Diffusion Transformers (DiTs), showing that attention can aggregate evidence from demonstrations to approximate posterior task weights, while diffusion enables sampling from the resulting predictive distribution. The authors further decompose the generation error into two components: the error in approximating the posterior-predictive distribution and the task uncertainty remaining after observing finite demonstrations. They establish finite-sample guarantees for learning the posterior-predictive distribution and prove that DiTs can achieve the minimax-optimal prompt-length rate.

Critical strengths

The paper provides an interesting theoretical insight on in-context generation by decomposing generation error into two components: the error in generating the posterior-predictive distribution (learning) and the task uncertainty remaining after observing finite demonstrations (demonstration may not be enough to determine what is the hidden task). This formulation explicitly accounts for uncertainty over latent tasks in in-context generation. Furthermore, the authors extend this framework to Diffusion Transformers and mathematically prove that they can achieve the minimax-optimal convergence rate (under some assumptions).

Critical weaknesses
  1. I believe some of the theoretical contributions by the authors overlap with Kunde et al. (2025) [1]. In particular, Kunde et al. (2025) have already shown that a softmax-attention Transformer can estimate the true posterior distribution from in-context demonstrations under specific assumptions. They also establish that the optimal parameter configuration minimises the asymptotic training objective. These results are closely related to the present paper's analysis of attention-based posterior inference and the learnability of predictive distributions. Although the present paper extends the setting to diffusion-based generation and provides finite-sample and minimax guarantees, the distinction from Kunde et al. (2025) is not sufficiently discussed. The authors should discuss or rescope the contributions they claim.

[1] Vishnu Teja Kunde et al. Transformers are Provably Optimal In-context Estimators for Wireless Communications. AISTATS, 2025.

  1. Unnecessarily complicated writing. Although the core concepts of the paper are relatively straightforward and intuitive, the writing frequently makes them more difficult to understand than necessary. For example, the abstract states:

We first show that the natural target to generate from finite demonstrations is not an output derived from estimating a single task, but rather a predictive distribution that captures the task uncertainty remaining after observing the demonstrations.

This could be expressed more simply as: Rather than selecting a single task and generating from its distribution, the model should generate from a predictive distribution that accounts for multiple plausible tasks.

Similar unnecessarily complex sentence structures and explanations appear throughout the paper, making it harder to follow the main ideas and theoretical contributions. A clearer and more concise presentation would substantially improve readability.

Questions for authors

How do the theoretical contributions differ from Kunde et al. (2025), particularly regarding attention-based posterior inference and the learnability of predictive distributions?

4 comments
AuthorsOpenReview ↗

Response to Reviewer GiBL

Thank you for the pointer to Kunde et al. (2025) and for the concrete writing suggestion. Both help us improve the paper's clarity.

W1/Q1. Relation to Kunde et al. (2025).

We agree that the two works are related, and we now credit Kunde et al. (2025). We study a different problem: they estimate a posterior over a finite set of symbols, whereas we aim to generate from an unknown continuous distribution that is only assumed to be Hölder smooth, so the model must approximate a score function and sample from it. Moreover, their results hold only in the limit of infinite prompt length, so they do not imply our finite-sample guarantees.

  • Posterior inference. Their Theorem 3 shows that, as the prompt length tends to infinity, attention recovers the symbol posterior given the true channel, which is analogous to our task-conditional distribution. Since the symbol set is finite, their analysis needs no smoothness or function approximation. We approximate the posterior predictive of a continuous output at a finite prompt length, for unknown H"older task families.
  • Learnability. Their Theorems 5–6 show that the limiting loss, as the prompt length and then the number of training prompts tend to infinity, is convex in the single trainable matrix of a one-layer model, and identify its minimizer. We bound the error of ERM over the whole trainable class from finitely many episodes.

Capturing these finite-sample and continuous-domain aspects requires substantially more involved technical machinery, combining nonparametric approximation theory with delicate uniform concentration arguments. In particular, three parts of our analysis do not appear in their asymptotic analysis:

  1. controlling how per-example approximation errors add up over the nn demonstrations (Prop. D.4);
  2. approximating the score with polynomially many parameters and bounding the complexity of the whole trainable class (Lemma E.1, Props. E.6–E.7, Thm. F.2);
  3. turning score accuracy into accuracy of finite-step diffusion samples (Cor. G.6), with a matching minimax lower bound (Thm. 3.2).

As you suggest, we have discussed and rescoped our claims. Section 1.1 compares the two works, and our second contribution now begins with "Softmax attention can compute posterior-type weights from in-context examples (e.g., Kunde et al., 2025)" before stating what is new.

W2. Writing.

We rewrote the abstract following your suggestion. The quoted passage is now:

We first identify the posterior predictive distribution, which weights the plausible tasks by their posterior probabilities, as a natural target for generation from finite demonstrations.

We also simplified several sentences in Sections 2 and 3, for example, the motivation of the pretraining target in Section 2.2 and the discussions after Theorems 3.1 and 3.2, and we will continue this throughout the paper.

Reference

  • Kunde et al. (2025). Transformers Are Provably Optimal In-Context Estimators for Wireless Communications. AISTATS 2025.
AuthorsOpenReview ↗

Gentle Reminder

Thank you again for your time and helpful feedback. As the discussion period is coming to a close, we wanted to follow up. Following your suggestions, we have expanded the comparison with Kunde et al. to explain the differences in problem setting and theoretical guarantees. We have also simplified the writing. We would appreciate your thoughts and would be happy to clarify any remaining concerns.

Best regards,

The Authors

Thank you for your response. I have updated my score accordingly.

Best, Reviewer GiBL

Thank you very much for your contribution to our paper!

Reviewer_4hT5OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the theoretical properties of Diffusion Transformers (DiTs) for in-context generation. The authors formulate the problem of generating samples from a task-specific distribution given a finite number of demonstrations and establish that the posterior-predictive distribution provides a suitable target for generation. They introduce a theoretical framework showing that DiTs can approximate this distribution through score estimation, using attention mechanisms to aggregate information from the demonstrations. Specifically, under Hölder regularity assumptions on the task distributions, the authors establish finite-sample guarantees for learning the posterior-predictive distribution, accounting for both the pretraining and diffusion sampling errors. Additionally, matching upper and lower bounds are established for the uncertainty arising from finite demonstrations, showing that DiTs can achieve the minimax-optimal rate of (O(\sqrt{\log n/n})) in task-conditional total variation distance. Numerical experiments illustrate the ability of DiTs to approximate posterior-predictive distributions and capture uncertainty from finite demonstrations.

Critical strengths

S1. The paper is well-written and provides a clear motivation for studying in-context generation from a theoretical perspective.
S2. The problem considered is interesting and relevant, as understanding the theoretical capabilities of Diffusion Transformers for in-context generation is an important research direction.
S3. The authors provide a strong theoretical analysis, establishing both finite-sample guarantees for learning the posterior-predictive distribution and matching upper and lower bounds for the uncertainty arising from finite demonstrations.
S4. An interesting component of the established theory is the decomposition of the generation error into two components: the error arising from pretraining and sampling, and the uncertainty remaining due to the finite number of demonstrations. This provides a useful interpretation of the different sources of error in in-context generation.
S5. The authors establish a minimax-optimal convergence rate with respect to the prompt length. Moreover, the theoretical guarantees account for the number of pretraining episodes and the number of diffusion sampling steps, providing a connection between statistical accuracy and computational resources.

Critical weaknesses

W1. The assumptions required for establishing the theoretical guarantees seem to be quite restrictive. In particular, Assumption 3.1 requires the task distributions to admit densities satisfying Hölder regularity and a uniform positive lower bound. Additionally, the latent task dimension is assumed to be fixed. While these assumptions seem reasonable for establishing the first theoretical guarantees, it would be interesting to understand whether they can be relaxed to cover more general distributions, especially those arising in practical applications of diffusion models.
W2. The theoretical guarantees are established for a simplified Transformer architecture. In particular, the construction relies on four attention blocks that implement an approximation of posterior averaging over a discretized latent task space. Although this is an interesting theoretical construction, it is not clear to what extent the analysis captures the behavior of the Diffusion Transformers used in practice. It would be good if the authors could discuss whether the theoretical guarantees can be extended to more standard DiT architectures and what the main obstacles would be.

Questions for authors

Q1. In Assumption 3.1, the task distributions are assumed to have densities that are uniformly bounded away from zero. How restrictive is this assumption for practical generative modeling problems? Is there a possibility of extending the results to distributions without a uniform positive lower bound, and if so, what are the main technical obstacles?
Q2. In Theorem 3.1, the authors establish that the required number of pretraining episodes and diffusion sampling steps scales polynomially with the desired accuracy. However, the polynomial dependence on the problem dimensions and regularity parameters is not immediately clear from the main theorem. Could the authors provide a more explicit discussion of these dependencies and their implications for practical applications? Q3. The minimax-optimal rate established in Theorem 3.2 is (O(\sqrt{\log n/n})). Is the logarithmic factor necessary for natural fixed task families, or does it mainly arise from considering the worst case over the general class of distributions? Under additional assumptions on the task family, would it be possible to establish an improved rate?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4hT5 (1/2)

Thank you for the positive assessment and the precise questions.

W1/Q1. How restrictive are the assumptions, and can the density floor be removed?

(1) Scope and practical restrictions. The assumptions are mild for full-dimensional data whose densities are bounded above and below, such as smooth mixtures with a uniform component or bounded exponential families (Appendix I). They exclude laws on lower-dimensional manifolds, which may matter for images.

(2) Extending the generation guarantee. The generation theorem uses the floor to bound log-likelihoods, in the latent discretization, and in the ratio readout (Remark H.1). For laws on lower-dimensional manifolds, there is an additional obstacle: such a law is at TV distance one from every absolutely continuous law, including the diffusion state at t0>0t_0>0. With our early-stopped sampler, an extension must therefore change either the metric, for example to a Wasserstein distance, or the final generation step (Section 4).

(3) The finite-prompt upper bound already holds without the density floor. That bound needs only a KL modulus KL(Gθ∣Gθ′)≤K∣θ−θ′∣22βθ\mathrm{KL}(G_\theta\\|G_{\theta'})\le K\\|\theta-\theta'\\|_2^{2\beta_\theta} (Lemma H.3), which the floor implies through KL≤χ2\mathrm{KL}\le\chi^2.

Hölder smoothness controls feature approximation and early stopping, and the sampling result alone needs only weaker local conditions (Cor. G.6). The fixed latent dimension dθd_\theta enters through the number of latent tasks (see Q2).

W2. Can the theoretical guarantees be extended to standard DiTs? What are the main obstacles?

Much of our analysis already applies beyond our construction: the error decomposition, the finite-prompt bounds, and the learning and sampling analyses do not depend on the architecture (General Response, point 1). Our construction also uses the same softmax attention and denoising objective as standard DiTs. We discuss two routes for extending our analysis to standard DiT:

  • Realizing our construction with standard blocks. Our construction uses hard attention masks, fixed coordinate updates, and one token per example. Standard attention and learned feedforward layers could approximate these computations. The main obstacles are preserving likelihood sums under layer normalization, retaining per-example information through patch tokens, and controlling how approximation errors accumulate across layers.

  • Analyzing standard DiTs directly. One could instead study the approximation ability of standard DiTs without our explicit construction, for example by pooling per-example features into a summary of the prompt. The key challenge is to show that both the model size and the logarithm of the covering number remain polynomial in the prompt length nn and the inverse accuracy ε−1\varepsilon^{-1}.

AuthorsOpenReview ↗

Response to Reviewer 4hT5 (2/2)

Q2. Explicit dependence on dimensions and regularity parameters.

With d=dstatd=d_{\mathrm{stat}}, b=β∧1b=\beta\wedge1, and r=dθ/βθr=d_\theta/\beta_\theta, and up to logarithms and constants of the fixed class, Appendix G.9 now gives:

  • latent tasks: J≲(1+(n+1)Hθ2/(p‾ε2))r/2J\lesssim\bigl(1+(n+1)H_\theta^2/(\underline p\varepsilon^2)\bigr)^{r/2};
  • trainable parameters: Nparam,n,εlat=O~(J2+Jε−(d+1)(b+3)/b2+J(nε−1−1/b)d/b)\mathsf{N}_{\mathrm{param},n,\varepsilon}^{\mathrm{lat}}=\widetilde{O}\bigl(J^2+J\varepsilon^{-(d+1)(b+3)/b^2}+J(n\varepsilon^{-1-1/b})^{d/b}\bigr);
  • episodes and sampling steps: N=O~(ε−2Nparam,n,εlat)N=\widetilde{O}(\varepsilon^{-2}\mathsf{N}_{\mathrm{param},n,\varepsilon}^{\mathrm{lat}}) and M=O~(ε−2)M=\widetilde{O}(\varepsilon^{-2}).

At ε=log⁡n/n\varepsilon=\sqrt{\log n/n}, this gives Nparam,n,εlat=O~(nκ)\mathsf{N}_{\mathrm{param},n,\varepsilon}^{\mathrm{lat}}=\widetilde{O}(n^{\kappa}), N=O~(nκ+1)N=\widetilde{O}(n^{\kappa+1}), and M=O~(n)M=\widetilde{O}(n), with κ=max⁡{2r, r+(d+1)(b+3)/(2b2)}\kappa=\max\{2r,\ r+(d+1)(b+3)/(2b^2)\} (κ=5\kappa=5 in the scalar case). The cost is polynomial in nn and ε−1\varepsilon^{-1} at fixed dimension. The exponent grows linearly with dd and does not improve for β>1\beta>1, so these worst-case bounds are not quantitative guidance for image-scale data.

Q3. Is the logarithmic factor necessary for fixed task families?

The logarithmic factor is not necessary for the fixed families we analyze, but it is unavoidable uniformly over the Hölder task class in Theorem 3.2.

  • Improved rates for fixed families. For Example I.1, we prove E[TV(P(⋅∣Cn),PΘ)]≤2/(πn)\mathbb E[\mathrm{TV}(P(\cdot\mid C_n),P_\Theta)]\le 2/(\pi\sqrt n) for every prior (Appendix I.3). More generally, Proposition I.1 gives a log-free O(n−1/2)O(n^{-1/2}) rate for sample-only families with finitely many fixed, linearly independent density components, including mixtures with fixed components. For such families that also satisfy Assumption 3.1, a pretrained DiT attains this rate with sufficient resources (Theorem 3.1 with ε≍n−1/2\varepsilon\asymp n^{-1/2}).

  • Why the worst-case logarithm remains. In the uniform upper bound, Lemma H.3 controls the last prediction step by I(Θ;U1:n+1)/(n+1)I(\Theta;U_{1:n+1})/(n+1), where U1:n+1U_{1:n+1} are the first n+1n+1 examples. This term is O(log⁡n/n)O(\log n/n) uniformly over the class but can be loose for a fixed family. The matching lower bound (Appendix H.4) uses nn-dependent families with about log⁡n\log n weakly distinguishable perturbations, so the logarithm cannot be removed uniformly.

AuthorsOpenReview ↗

Gentle Reminder

Thank you again for your time and helpful feedback. As the discussion period is coming to a close, we wanted to follow up. Your questions helped us clarify our assumptions, the relation to standard DiTs, and the dependence on dimensions and smoothness. We also added log-free bounds for specific fixed task families. We would appreciate your thoughts and would be happy to clarify any remaining concerns.

Best regards,

The Authors

Thank you very much for your detailed response! I maintain my positive evaluation of the paper.

Regards,
Reviewer 4hT5

Reviewer_xgsKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper develops a statistical theory of in-context generation, where a pretrained model must generate an output distribution from finitely many demonstrations of an unobserved task. It distinguishes approximation of the posterior-predictive distribution from uncertainty about the realized task that remains after observing the prompt.

Under compact, finite-dimensional latent-task parameterization, Hölder regularity, and uniformly positive task densities, the authors construct a diffusion Transformer that approximates the posterior-predictive score. They combine representation, finite-episode score-learning, and finite-step diffusion-sampling guarantees. They also establish matching upper and worst-case lower bounds of order (sqrt{log n/n}) in expected task-conditional total variation, showing that the constructed learner attains the optimal prompt-length rate given sufficient resources and approximate empirical-risk minimization.

Critical strengths
  1. The paper goes beyond showing that a Transformer can represent a known posterior-predictive rule. It constructs a common model class independent of the unknown task family and prior, controls learning from independent episodes without task labels, and translates integrated score error into finite-step generative TV error. The episode-level KL discretization argument is particularly useful: it avoids requiring uniform posterior approximation over all possible prompts. This makes the polynomial-resource construction more substantive than a generic universal-approximation argument.

  2. The matching upper and lower bounds provide a meaningful benchmark independent of the neural architecture. The lower bound remains applicable even when the generator knows the environment and has unrestricted computation, clearly separating information limitations from learning and sampling errors. The explicit distinction between worst-case families and fixed-family behavior strengthens the result. The paper also handles both sample-only and input–output prompts within a unified framework.

Critical weaknesses
  1. The optimality result concerns the dependence on prompt length under sufficient resources. It does not establish optimal pretraining complexity, computational complexity, or sampling-step complexity, nor does it show that diffusion Transformers uniquely achieve the rate.

  2. The construction uses specialized latent-task tokens, masks, coordinate maps, and a fixed ratio readout. The architecture is indexed by n and approximation accuracy; it does not establish that one fixed pretrained model attains the rate across growing prompt lengths. Approximate empirical-risk minimization is assumed without an efficient optimization guarantee.

  3. Appendix J computes the “exact” posterior predictive using simulator-side r1:nr_{1:n} and c1:nc_{1:n}, while the learned model receives rendered context images. Unless rendering preserves these variables sufficiently, the analytical reference is a posterior conditional on richer information than the model observes. Apparent calibration error may therefore combine model error with information lost through rendering.

Questions for authors
  1. Is the analytical reference in Appendix J exactly the posterior predictive conditional on the rendered image context? Can rir_i and cic_i be recovered from those images without relevant ambiguity? If not, how large is the difference between the simulator-conditioned and image-conditioned predictive distributions?

  2. Which part of the result depends essentially on the diffusion Transformer construction, beyond providing a realizable conditional score class? Please distinguish the architecture-independent prompt lower bound from the architecture-specific approximation and learning guarantees.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xgsK

Thank you for the careful reading and the positive assessment.

W1. Scope of the optimality result.

As you note, our optimality concerns the prompt length. We focus on it because, once a model is deployed, only the prompt changes: pretraining is done once by the model developer, whereas each user adapts the fixed model to a new task through demonstrations, which are often costly to collect (for example, robot demonstrations) and limited by the context length. The optimal prompt-length rate gives the best worst-case accuracy that any generator can achieve from nn demonstrations, and hence how many demonstrations a target accuracy requires. Prior ICL theory studies the same question (e.g., Bai et al., 2023; Kim et al., 2024).

We agree pretraining complexity also matters. We revised the abstract and introduction to state that we claim no optimality in model size, data, computation, or sampling steps, nor that only DiTs attain this rate. On pretraining complexity, we now also prove that pretraining is necessary to attain the optimal prompt-length rate uniformly over our class (Section 3.3; Proposition H.5). Without pretraining, the worst-case rate is no faster than n−β/(2β+d)n^{-\beta/(2\beta+d)}, and attaining log⁡n/n\sqrt{\log n/n} requires at least of order nd/(2β)n^{d/(2\beta)} pretraining episodes up to logarithms. This is below our sufficient number, so optimal pretraining complexity remains open.

W2. Architecture, fixed models, and optimization.

Such simplifications are common in statistical learning theory for Transformers (e.g., Bai et al., 2023; Kim et al., 2024), and Remark E.1 lists how our class differs from standard DiTs.

The fixed parts of the construction depend only on (n,ε)(n,\varepsilon) and the known dimensions and regularity bounds, never on the task family or prior. All attention matrices, prior logits, and the output coefficients of the feature networks are learned, and the learning guarantee covers this entire class.

Our construction grows with nn because per-example approximation errors add up over the nn examples, so the latent grid and the likelihood accuracy are refined as nn grows (Prop. D.4, Eq. (143)). Nonparametric ICL analyses such as Kim et al. (2024) also let the model grow with nn. A guarantee for a single model across prompt lengths remains open. In practice, each of our experiments trains one model on mixed context lengths (e.g., Figure 7, right).

Like Bai et al. (2023) and Kim et al. (2024), we analyze the statistical accuracy of empirical risk minimization rather than optimization dynamics. Our guarantee allows an optimization gap εopt≲ε2\varepsilon_{\mathrm{opt}} \lesssim \varepsilon^2. Joint statistical and optimization guarantees are an important direction for future work (see Section 4).

W3/Q1. Can the analytical reference be recovered from the rendered image context?

We confirmed that the analytical posterior can be reconstructed almost exactly from the images observed by the DiT. Across 64 tasks and 4,096 image pairs, the maximum reconstruction errors for rir_i and cic_i were 5.35×10−75.35\times10^{-7} and 7.11×10−87.11\times10^{-8}, respectively. Moreover, across all 448 evaluation conditions, recomputing the reference from the recovered coordinates changed it by at most 1.19×10−61.19\times10^{-6} in right-mode probability and 6.55×10−76.55\times10^{-7} in W1W_1, four orders of magnitude below the DiT errors (0.0160.016 and 0.0410.041). Thus, although the reference in Appendix J is computed from the simulator coordinates, the information lost in rendering is negligible for our evaluation (see Appendix J.1).

Q2. Which parts depend on the DiT construction?

Only the score class. Our analysis uses two properties of this class: it contains an accurate score model, and the logarithm of its covering number is polynomial in nn and ε−1\varepsilon^{-1}. The covering bound controls the finite-episode estimation term, and the approximation error enters separately.

  • Architecture-independent results. Everything else holds for any architecture, under the stated assumptions: the error decomposition (Eq. (3)), both bounds of Theorem 3.2, the bDSM identity (Prop. C.1), the episode-level KL discretization (Prop. D.4), the learning theorem (Thm. F.2), and the sampling analysis (Cor. G.6).

  • Architecture-specific results. The DiT-specific part is a score class with accurate approximation and polynomial complexity as the prompt grows (Lemma E.1 and Props. E.6–E.7).

  • Role of the construction. The per-example networks have an input dimension that does not grow with nn, and attention pools their outputs. Approximating the prompt-to-score map as an unstructured function of the concatenated prompt would give bounds exponential in nn.

References

  • Bai et al. (2023). Transformers as Statisticians: Provable In-Context Learning with In-Context Algorithm Selection. NeurIPS 2023.
  • Kim et al. (2024). Transformers are Minimax Optimal Nonparametric In-Context Learners. NeurIPS 2024.
AuthorsOpenReview ↗

Gentle Reminder

Thank you again for your time and helpful feedback. As the discussion period is coming to a close, we wanted to follow up. Your comments helped us clarify the scope of our guarantees. We also added a lower bound on pretraining size and checked coordinate recovery from the rendered images. We would appreciate your thoughts and would be happy to clarify any remaining concerns.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revisions

All changes are marked in blue in the updated manuscript.

  • Section 3.3 and Appendix H.6 (Proposition H.5): pretraining is necessary to attain the optimal prompt-length rate uniformly over our model class (xgsK).
  • Appendix J.1: a reconstruction check relating the analytical reference to the rendered images (xgsK).
  • Appendix J.3: Fashion-MNIST experiments on pretraining, context length, and context composition (thgp).
  • Abstract and introduction: clearer motivation of the posterior-predictive target and the scope of the optimality claim (GiBL, xgsK); Section 1.1 adds a comparison with Kunde et al. (2025) (GiBL).
  • Remarks E.1–E.2: relation to standard DiTs and steps toward guarantees for them (xgsK, 4hT5, thgp).
  • Appendix G.9: explicit dependence on dimensions, regularity, and accuracy (4hT5).
  • Remark H.1 and Appendix I.3 (Proposition I.1): role of the density floor, and log-free rates for fixed families (4hT5).
  • Section 4: extensions and open questions (xgsK, 4hT5, thgp).
AuthorsOpenReview ↗

General Response

We thank all reviewers for their detailed and thoughtful feedback. The reviewers found that the paper "goes beyond showing that a Transformer can represent a known posterior-predictive rule" and that its matching bounds "provide a meaningful benchmark independent of the neural architecture" (xgsK), with a "strong" analysis (4hT5), an "interesting theoretical insight" in the error decomposition (GiBL), and "a clear conceptual message" (thgp).

As the author discussion draws to a close, we summarize four questions that were raised across the reviews. We answer them briefly here and in detail in the individual responses.

1. How does the analyzed DiT relate to standard DiTs? (xgsK W2/Q2, 4hT5 W2, thgp W1)

The core of our theory is a construction showing that attention can form posterior weights over tasks from the demonstrations and combine them into the posterior predictive score, with model size polynomial in the prompt length and the inverse accuracy (Lemma E.1, Props. E.6–E.7). This part is specific to our simplified DiT, which uses the same softmax attention and denoising objective as standard DiTs. The remaining analysis, namely the error decomposition, the finite-prompt bounds, and the score-learning and sampling analyses, does not depend on the architecture. We have added Remark E.1 to lists the differences from standard DiTs, and Remark E.2 describes two routes toward extending our analysis to standard DiT.

2. Do the experiments support the theory? (xgsK W3/Q1, thgp W2/Q2)

We added a Fashion-MNIST experiment in which the category-level posterior predictive is known exactly. The category frequencies generated by a standard DiT approach it as pretraining grows (TV from 0.5090.509 to 0.0560.056, Appendix J.3). For the rendered-shape experiment, we verified that recomputing the analytical reference from the images observed by the model changes it by less than 2×10−62\times10^{-6} (Appendix J.1).

3. What is new relative to Kunde et al. (2025)? (GiBL W1/Q1)

The two works share one idea, that attention can compute posterior weights, and our results do not follow from theirs. Kunde et al. (2025) prove this idea for estimating a posterior over a finite set of symbols, in the limit of infinite prompt length. We generate from an unknown continuous distribution, where posterior weights alone are not enough: the guarantee also requires approximating the score, learning it from finitely many pretraining episodes, and sampling in finitely many steps. We prove all of these at a finite prompt length, together with a matching lower bound. Capturing these finite-sample and continuous-domain aspects requires substantially more involved technical machinery, combining nonparametric approximation theory with delicate uniform concentration arguments. Section 1.1 now compares the two works and credits the shared idea.

4. What does "optimal" refer to? (xgsK W1)

It refers to the rate in the prompt length, given sufficient pretraining and sampling resources. We focus on the prompt length because, once a model is deployed, only the prompt changes. The revised introduction states that we claim no optimal dependence on the other resources. We also added Proposition H.5, which shows that pretraining is necessary to attain this rate uniformly over our model class.

Reference: Kunde et al. (2025). Transformers Are Provably Optimal In-Context Estimators for Wireless Communications. AISTATS 2025.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.