- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This is a theoretical paper on in-context generation with Diffusion Transformers (DiT). The authors argue that the right target is to generate samples from the posterior-predictive distribution rather than a single output. They analyze a simplified prompt-conditioned DiT and show that it can learn and sample from this distribution from pretraining episodes, without task labels, using polynomial model size, data, and sampling steps. They further prove that the error due to a finite prompt scales in terms of the number of demonstrations n, with a matching lower bound, so the trained DiT attains the minimax-optimal rate in prompt length. A small-scale synthetic image experiment illustrates the predicted behavior (bimodal predictive distribution under short contexts, contracting as context grows). Overall this is a thorough theoretical contribution with a clear conceptual message.
Critical strengths
• Clear formulation of in-context generation as sampling from the posterior-predictive distribution. The main message — that in-context inference is Bayesian posterior prediction and uncertainty should be preserved — is already present in prior work the paper cites; the contribution is extending it to generation and proving it rigorously. • Comprehensive theoretical derivation covering the full pipeline from model capacity through learning from data to sampling, with a matching lower bound
Critical weaknesses
• The analyzed model differs substantially from DiTs used in practice. The experiment uses a conventional DiT rather than the analyzed one, so the connection between the theorems and practical DiTs is asserted rather than demonstrated. • The experiment shows the qualitative behavior (bimodality, contraction with more demonstrations) but do not directly evaluate the theoretical error rates for a learned DiT.
Questions for authors
• Does equation (10) give practical guidance? Appendix G.9 gives N = Õ(n⁶) and M = Õ(n) for the scalar case, while the experiment used about 10⁶ episodes at n ≤ 64. Can the authors reconcile this gap? • The empirical evaluation is limited to controlled synthetic settings. While these examples illustrate the theoretical results, experiments on more realistic in-context generation tasks would strengthen the practical relevance of the work.
5 comments
Response to Reviewer thgp (2/2)
Q1. Does Eq. (10) give practical guidance, and why are the experimental budgets much smaller?
It gives qualitative guidance. Eq. (9) bounds the posterior-predictive error by plus an estimation term and a sampling term . Here is the number of trainable parameters, the number of pretraining episodes, and the number of sampling steps. Eq. (10) gives the resulting sufficient and for error . More pretraining or sampling does not reduce the separate finite-prompt term. Appendix K turns this into diagnostics: with the checkpoint fixed, increase the sampler steps. If the output still differs from the posterior predictive, improve training, capacity, or use of context. Once these errors are small, more informative demonstrations help most.
The experimental budgets are far below because that bound covers the worst case over the Hölder class, for which the scalar construction uses representative tasks. Our rendered-shape family is much simpler: its posterior depends on the context only through five sums, and its predictive distribution is a two-component Gaussian mixture. Parameter-count bounds are also loose for overparameterized networks, and the step bound concerns the randomized SDE sampler rather than DDIM.
Q2. More realistic in-context generation tasks.
The Fashion-MNIST experiments move from rendered shapes to real images while keeping the category-level posterior predictive known, so that calibration can be checked directly.
-
Task. Each episode draws category proportions from and samples context and target images from that mixture.
-
Result. When we vary the category composition of a 64-image context (Figure 8, all ten category pairs), the generated frequency of the second category increases at every tested step. Although the task structure is still controlled, we confirmed that the results are consistent with our theory.
Gentle Reminder
Thank you again for your time and helpful feedback. As the discussion period is coming to a close, we wanted to follow up. We have expanded the discussion of how our construction relates to practical DiTs and added Fashion-MNIST experiments that measure generation error as pretraining and context length vary. We would appreciate your thoughts and would be happy to clarify any remaining concerns.
Best regards,
The Authors
Response to Reviewer thgp (1/2)
Thank you for the positive assessment and the suggestions. We address your concerns and have added experiments on Fashion-MNIST (Appendix J.3).
We agree that our theorems apply to a simplified DiT rather than to conventional DiTs, and we have clarified this difference in the revised manuscript (Remark E.1). Our theory still connects to practical DiTs in three ways.
Architecture-independent results. The error decomposition and the finite-prompt bounds hold for any generator, including practical DiTs, and the learning and sampling analyses apply to any score class with suitable approximation and complexity bounds. Our experiments measure the quantities in these results: the posterior-predictive error and the finite-prompt term.
Relevance to practical DiT experiment. Our experiments support the main qualitative predictions of the theory. While our theory focuses on the stylized architecture, we empirically observe similar qualitative trends in a standard DiT, suggesting that these phenomena are not specific to the simplified model.
Extension to practical DiT's theory. While our work represents a starting point, we believe this theoretical framework has the potential to be extended to more standard DiT architectures. We have added two possible directions in Remark E.2: extending our explicit construction to standard DiTs or directly analyzing their approximation capabilities. The main challenge is to establish polynomial approximation and complexity bounds while accounting for layer normalization, patch tokenization, and learned MLPs.
We now measure category-level generation error as pretraining and prompt length vary, and compare it with the exact finite-prompt benchmark, which is known at the category level (Fashion-MNIST, 24 test tasks). These experiments examine a fixed family, while the minimax rate concerns the worst case over the full class.
Each image belongs to one category and is drawn independently of the task given its category, so the label-based reference equals the image-based one at the category level. For this category mixture, an argument analogous to Proposition I.1 gives a finite-prompt term of order n−1/2 without a logarithm. We used 100 DDIM steps and did not vary the number of sampling steps.