acceptodds
Under review as a conference paper at ICLR 2027

Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation

Abstract

Text-to-video generation has advanced significantly over the past five years through scaling of model size, data, and compute. Unlike model architecture, training data is often underexplored. Real-world data curation is complex and non-trivial, involving clip selection from raw videos and captioning to create video-text pairs for learning text-to-video mappings. We study how data distribution and caption quality impact text-to-video models. To enable controlled experiments, we introduce Moving Alphabet, a procedural testbed that renders letters with varying fonts, colors, sizes, and positions, moving in different directions and speeds against a black background. This design allows precise control over data distribution and caption quality by corrupting ground-truth metadata. Our experiments yield three findings: a) a diverse and balanced distribution of video content and duration is critical for generalization; b) caption quality significantly affects both model performance and training efficiency, suggesting that text-to-video models are bounded by video understanding capabilities; and c) recovery through clean-data fine-tuning depends on the severity and type of caption corruption. We believe these insights can inform the development of large-scale text-to-video models, and we advocate for greater attention to the science of pre-training data.

Primary area:
generative models
Keywords:
Video Generation, Training Data Curation, Controlled Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_YosgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a procedural testbed of letters moving on a black background, where every attribute and its caption are generated from known metadata. An 800M MMDiT is trained from scratch on controlled mixtures to study three questions: how to mix scene complexity and clip duration, how caption precision and recall affect quality and training compute, and whether clean-data finetuning recovers from corrupted captions.

Critical strengths
  1. The testbed makes questions measurable that are confounded on real data.
  2. The precision-versus-recall asymmetry is specific and useful.
Critical weaknesses
  1. The main results appear to be single runs with no error bars, yet several conclusions rest on small gaps. Mix Equal beats Heavy-2s by 0.3 dB on average length and Pure 1L by 0.4 dB on 1L. The paper's own appendix shows this setup is seed-sensitive: in the drop-one study over 4 seeds, size bias varies across KL 0.003–0.056 and each seed favors a different color.

  2. The findings are never checked against real video, yet are stated as curation guidance. The domain is letters on black at 256×256, captions are templates with exact angles and coordinates, and corruption replaces attributes with uniformly random wrong values. Real captioner errors are systematic and correlated, and real recall is ill-defined.

Overall, I think this paper is more like an experimental report, and the core findings seems quite obvious, without much new insights.

Questions for authors
  1. Can you repeat the headline conditions in Tables 1 and 2 over several seeds and report variance? Which comparisons remain significant, particularly Mix Equal versus Heavy-2s and versus Pure 1L?

  2. Does the precision-over-recall ordering hold under correlated, plausible corruptions, such as swapping an attribute to a nearby value, rather than uniformly random replacement?

  3. Could a small real-data study, for instance corrupting captions on an existing video corpus, test whether any finding transfers outside the testbed?

Reviewer_WVRfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This work conducted a controlled study of how quality of training dataset affect performance of video generation models. Specifically, it uses synthetic video featuring moving and bouncing letters, varies the complexity, length, and quality of textual description and studied the performance impact of these factors. The paper analyzed the results and gave some insights on how dataset quality affects model performance.

Critical strengths
  1. The motivation of this research is sound, understanding how different dataset quality factors affect performance is both useful in practice and might provide insights for future research.
  2. The paper is overall easy to follow and understand.
Critical weaknesses

The results presented in the paper are unsurprising and do provide much new insights:

  1. Data with more variety naturally leads to better generalization ability, this is a trivial conclusion in modern deep learning research.
  2. "Corrupted textual description leads to performance reduction" is also trivial. "Precision is more important than recall" is slightly more interesting, but it is still something to be expected from modern generative models, especially diffusion models: these models are often trained to simultaneously retain conditional and unconditional generative capabilities (and this is mandatory if using classifier-free-guidance, a.k.a., CFG). Omitting part of the caption simply degrades the task from conditional- to unconditional-generation, whereas introducing wrong information natually corrupts the dataset much more severely.

Additionally, the design of the study itself raises other concerns:

  1. The synthetic data does not align well with real data: The synthesized videos are extremely simple, and the accompanying caption captures almost all of its information in natural language. As shown in the caption of Fig. 1, the prompt specifies the coordinates and rotations in a numerically precise manner. In practice, however, caption and user input would never capture all information in real videos. How well do the findings translate to real scenarios is thus questionable.
  2. The pixel-level evaluation protocol does not align with real needs of video generation models, either: As is mentioned in W3, real captions are under-specifying. User would seldom expect the model to output something pixel-perfect w.r.t. a prescribed ground truth. In other words, for any given caption, there exist multiple output videos that are considered valid, so long as they follow the essential information in the instructions. A pixel-level evaluation protocol does not align with this, the practical value of the findings is thus unclear.

Apart from that, there are other, uncategorized yet prominent weaknesses:

  1. The introduction to existing works contains very inaccurate information: At line 034, diffusion models are introduced as an example of "model and architectural innovations". This is completely wrong: diffusion model, by itself, has nothing to do with model architecture. What was proposed as "diffusion model" is a novel way to formulate probabilistic generation problems and a corresponding training strategy.
Questions for authors
  1. (W1, W2) The novelty of the reported findings is unclear. Please address the following question: Which conclusions go beyond well-known and commonly acknowledge facts? How are the provided insights significant and useful?
  2. (W3) The alignment between the synthetic setting and real scenarios is unclear. Please address the following questions: Would the main findings still hold when captions are under-specifying, as in real use? Can the authors validate the conclusions on real videos or more realistic synthetic data?
  3. (W4) The practical value of the pixel-level evaluation protocol is unclear. Please address the following questions: Since real captions are under-specifying and multiple outputs can be valid, why is a pixel-level protocol appropriate for measuring generation performance? How well does the results under this protocol translate to real scenarios? Can the authors add semantic or perceptual metrics, or human evaluation, to test whether the conclusions still hold in a more realistice setting (e.g., the input caption is under-specifying)?
Reviewer_owMJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Moving Alphabet, a procedural video testbed for investigating how training-data composition and caption quality affect text-to-video generation. Videos contain one to three moving letters with controlled appearance and motion attributes, allowing the authors to vary scene complexity, clip duration, and caption precision and recall. The reported findings favor balanced coverage of scene complexity and duration, accurate captions, and investment in caption quality during pretraining. Clean-data fine-tuning provides recovery that depends on the type and severity of corruption within the tested budget.

Critical strengths
  1. The procedural testbed enables useful control over training-data factors. The known scene metadata allows the authors to change caption correctness and completeness while holding the underlying videos fixed, and to construct explicit mixtures of complexity and duration. This reduces confounding and provides a useful experimental setting for studying data curation. The 35-condition corruption grid and the metric calibration in Appendix make the comparisons more systematic and interpretable.

  2. Some results go beyond a simple demonstration of distribution coverage. In Table 1, balanced mixing achieves 12.9 dB foreground PSNR on three-letter evaluation, compared with 10.1 dB for the three-letter specialist. The study also quantifies training-efficiency penalties and examines clean-data recovery. These observations offer concrete starting points for investigating transfer across scene complexities and the cost of imperfect supervision. They are positive evidence for the paper's potential contribution.

Critical weaknesses
  1. The empirical observations are not developed into a sufficiently distinctive conceptual contribution. The headline conclusions, including the benefits of diverse training data and accurate captions, are largely intuitive. The controlled testbed makes these effects measurable, but the manuscript provides limited evidence explaining why they occur or establishing the conditions under which they hold. The most interesting finding, that mixed-complexity training can outperform a specialist on the specialist’s own distribution, is accompanied by a compositional-transfer hypothesis that is not directly tested against alternative explanations such as optimization or regularization effects (Section 4.1). Consequently, the study remains largely descriptive, and the depth of new understanding is insufficiently established. This is my main concern regarding the contribution’s suitability for acceptance in its current form.

  2. The most informative anomalies require stronger validation and interpretation. Section 5.1 reports a sharp foreground-PSNR drop from 12.4 to 6.6 dB when caption precision decreases from 0.99 to 0.95, but does not report variability across independent training runs for this comparison. Without such evidence, it is difficult to assess whether this apparent threshold is a reproducible phenomenon. The precision–recall interpretation also leaves a potential confound unresolved. Appendix H shows that an omitted direction remains recoverable from start/end coordinates and bounce information. Depending on how the main corruption procedure handles these dependent fields, omission may preserve information through redundant descriptions, whereas replacement may introduce contradictory constraints. The manuscript therefore does not sufficiently separate the effects of incorrect supervision, information redundancy, and inconsistent conditioning. This limits the strength of the conclusion that caption correctness is generally more important than completeness.

  3. The experimental presentation lacks a consolidated overview of the study design. The paper investigates content complexity, video duration, caption corruption, and recovery through clean-data fine-tuning, but their relationships are described across separate sections without an overview of the experimental pipeline. Figure 1 illustrates a dataset sample rather than the organization of the study. Although the individual protocols are provided, readers must piece together which experiments use the 150K-video dataset or the 50K-video subset, which factors are varied or held fixed, and how the corrupted models feed into the fine-tuning experiments. This fragmented presentation makes the experimental logic harder to follow and obscures how the individual observations support the overall conclusions. An experimental overview figure would substantially improve clarity.

Questions for authors
  1. What is the principal new insight, and what evidence supports its explanation? Which specific result constitutes the main scientific contribution beyond quantifying the expected benefits of diverse data and accurate captions? In particular, what evidence supports compositional transfer as the reason mixed-complexity training outperforms the specialist on its own distribution? Can the authors distinguish this explanation from optimization or regularization effects induced by changing the training mixture?

  2. How robust is the sharp precision-related degradation, and how are dependencies between caption fields handled? Is the drop from 12.4 to 6.6 dB between precision levels 0.99 and 0.95 reproducible across independent training runs? Please also clarify how corruption is applied to dependent fields such as direction, start/end coordinates, and bounce events. When an attribute is omitted or replaced, are the related fields retained, removed, or updated consistently? What evidence distinguishes the reported precision–recall asymmetry from the effects of redundant information remaining after omission and contradictions introduced by replacement?

  3. Could the authors provide an overview figure connecting the experimental components? Please show how procedural data generation, complexity and duration mixtures, caption corruption, model training, clean-data fine-tuning, and evaluation fit together. The overview should identify the data subsets and checkpoints used in each study, the factors being varied, and the settings held fixed, particularly when moving from the 150K-video distribution experiments to the 50K-video corruption experiments and subsequent fine-tuning. This would make the organization and logic of the controlled study much easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.