- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a procedural testbed of letters moving on a black background, where every attribute and its caption are generated from known metadata. An 800M MMDiT is trained from scratch on controlled mixtures to study three questions: how to mix scene complexity and clip duration, how caption precision and recall affect quality and training compute, and whether clean-data finetuning recovers from corrupted captions.
Critical strengths
- The testbed makes questions measurable that are confounded on real data.
- The precision-versus-recall asymmetry is specific and useful.
Critical weaknesses
-
The main results appear to be single runs with no error bars, yet several conclusions rest on small gaps. Mix Equal beats Heavy-2s by 0.3 dB on average length and Pure 1L by 0.4 dB on 1L. The paper's own appendix shows this setup is seed-sensitive: in the drop-one study over 4 seeds, size bias varies across KL 0.003–0.056 and each seed favors a different color.
-
The findings are never checked against real video, yet are stated as curation guidance. The domain is letters on black at 256×256, captions are templates with exact angles and coordinates, and corruption replaces attributes with uniformly random wrong values. Real captioner errors are systematic and correlated, and real recall is ill-defined.
Overall, I think this paper is more like an experimental report, and the core findings seems quite obvious, without much new insights.
Questions for authors
-
Can you repeat the headline conditions in Tables 1 and 2 over several seeds and report variance? Which comparisons remain significant, particularly Mix Equal versus Heavy-2s and versus Pure 1L?
-
Does the precision-over-recall ordering hold under correlated, plausible corruptions, such as swapping an attribute to a nearby value, rather than uniformly random replacement?
-
Could a small real-data study, for instance corrupting captions on an existing video corpus, test whether any finding transfers outside the testbed?