acceptodds
Under review as a conference paper at ICLR 2027

Blue Mage: Tracing Learning Dynamics of Web-Scale Text-to-Image Flow Models

Abstract

We introduce Blue Mage, an open training record for web-scale text-to-image flow pretraining, in which every saved generator is bound to the updates that produced it. Pythia and OLMo made language-model pretraining a shared experimental object; text-to-image pretraining still reaches the public as its endpoint, rarely as the states that led to it. Beyond a data order, a flow-matching record has to fix what each batch slot draws: a sample, a flow time, a noise draw, and a caption decision. With the corpus distributed as image bytes, Blue Mage fixes all four in a closed-form plan before training, so the batch behind any update and the exposure of any concept before any saved state are recomputed from the plan; it repeats curated canary images against loss-masked twins and keeps resumable training states. We train MMDiT of 0.5B, 2.0B, and 3.9B parameters from scratch on nested corpora of 1M, 10M, and 100M images from the GPIC dataset. Following each run through its saved states, we ask when an attribute-entity pair is learned and when a repeated image becomes retrievable. Exposure sets how well a pair is rendered; training progress sets when. A familiar entity becomes easier to render and harder to control. Incorrect bindings rise with correct ones, peak with their exposure, and recede. Retrieval takes more updates yet fewer repetitions in a larger corpus at the highest dose, and varied captions weaken retrieval through the trained caption while strengthening it through captions never trained on. We will release the code, plans, exposure records, and saved states, and hope they serve open research on generative models throughout their training.

Primary area:
generative models
Keywords:
Text-to-image, Reproduciblity

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_HbScOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Blue Mage, a traceable training record system for text-to-image flow models. It records the samples, random variables, and conditioning information for each training slot, and stores concept exposure records, dense checkpoints, and recoverable training states. The authors train 0.5B to 3.9B MMDiT models on datasets ranging from 1M to 100M images, and use the full training trajectories to study attribute binding and memorization dynamics, including the relationship between exposure and learning, and how repeated samples and caption variation affect instance retrieval.

Critical strengths

1、The traceable training record itself is a useful experimental infrastructure. It links saved checkpoints with reconstructable per-slot training events, concept exposure histories, and recoverable optimizer states. Compared with common releases of open text-to-image models, this is clearly more complete and has the potential to support many future studies of training dynamics. 2、The memorization experiments are carefully controlled. The repeated, varied-caption, and loss-masked runs share the same initialization and training data stream, and differ only in the specified canary treatment. Compared with directly comparing independently trained models, this matched design makes the conclusions about memorization easier to interpret and more convincing.

Critical weaknesses

1、The evidence for generalizing training dynamics across runs is still limited. Most learning-dynamics conclusions are based on a single training realization for each condition. A large number of prompts, generation seeds, and canaries can characterize one training trajectory well, but cannot measure training-level variance caused by different initializations or data orders. This is especially important because a central question of the paper is when these phenomena emerge during training. 2、The binding conclusions rely heavily on an evaluator that may be sensitive to wording. The main binding analysis uses a single VLM judge, while the appendix shows that changing only the wording of the question can noticeably change entity-presence judgments on the same images. Since acquisition onset is defined by a threshold on this judge score, the robustness of some training-trajectory conclusions to the evaluator remains unclear.

Questions for authors

1、Have the main trajectory-level findings, such as the weak relationship between exposure and onset and the rise and subsequent decline of crossed binding, been reproduced in an independent training run? Replication even at a smaller model or data scale would be helpful. 2、How robust are the binding results to the choice of evaluator? Results from a second calibrated VLM evaluator or a small set of human annotations would help validate these conclusions.

Reviewer_3j3XOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper introduces Blue Mage, a training record for studying how text-to-image flow models learn concepts and memorize images during training. Blue Mage links each saved model state to its training history, allowing researchers to reconstruct concept exposure and resume training for controlled experiments. Using MMDiT models trained from scratch on GPIC subsets ranging from 1M to 100M images, the authors investigate when attribute–entity combinations are learned and how training progress, image repetition, and caption variation affect the reproduction of training images.

Critical strengths
  1. Blue Mage enables controlled studies of learning dynamics through traceable training histories and resumable checkpoints.
  2. The paired experiments show that caption variation can shift retrieval to unseen prompts, providing useful insights for evaluating memorization.
Critical weaknesses
  1. The paper is poorly written, with convoluted phrasing, overly compressed statements, and unclear transitions that make its main arguments difficult to follow. e.g. “training progress sets when” and “peak with their exposure” (Abstract, lines 23–26), as well as the densely packed discussion in the Introduction (lines 45–50), where the intended meaning and connections between statements require substantial interpretation.

  2. The paper defines the onset of concept learning using a generation success threshold (Section 3.3). However, it does not sufficiently validate the robustness of this onset to changes in prompts and evaluators, leaving its interpretation as a measure of concept acquisition uncertain.

Questions for authors
  1. What precisely do “training progress sets when” and “peak with their exposure” mean, and which measurements and results in Sections 5.2 and 5.4 support these statements?

  2. Do the estimated concept learning onset times and the main trends in Section 5.2 remain consistent across different prompts, independent evaluators, and alternative success thresholds?

Reviewer_ZXn4OpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Blue Mage, a traceable record of text-to-image pretraining. A closed-form OrderPlan fixes, before training, the sample, flow time, noise draw, caption, and loss mask assigned to every update and batch slot, so the training history of any saved state and the exposure of any concept can be recomputed from the plan rather than from a log written during training. Curated canaries repeated at assigned doses, each paired with a loss-masked twin on the identical stream, make repetition a controlled variable. The authors train MMDiTs of 0.5B, 2.0B, and 3.9B parameters from scratch on nested corpora of 1M, 10M, and 100M images, save 40–117 intermediate states per run, and release code, plans, exposure records, checkpoints, and resumable training states.

The reported findings are: exposure sets the level of an attribute–entity pair while training progress sets its timing; frequent entities are rendered more reliably but bound to requested attributes less reliably; a frequent crossed pair rises and then recedes during training; a larger corpus requires more updates but fewer repetitions for retrieval; and varied captions relocate retrieval from the original caption to untrained captions rather than removing the memorized image.

Critical strengths
  1. The traceable training record is a real contribution. The OrderPlan fixes every stochastic choice per slot in closed form, so any saved state's stream and any concept's exposure can be recomputed without relying on a training-time log. This turns the "model state–training history" binding from an approximate audit into a queryable object, which is directly useful for work on memorization, data attribution, and reproducibility. This is the consideration that most strongly supports acceptance.

  2. The canary and masked-twin design makes repetition an intervention. Holding plan, initialization, and all other slots fixed while changing only the canary-slot loss mask or caption allows repetition effects to be read at every saved state, and allows withdrawal/restoration experiments from a shared state. This is the most convincing experimental part of the paper and makes Sections 5.5–5.7 more credible than a purely correlational analysis would be.

Critical weaknesses
  1. The central natural-exposure results are associational, and each cell has a single training run. The claims in Sections 5.2–5.4 that exposure sets a pair's level while training sets its timing, that frequent entities bind harder, and that crossed pairs rise and recede all come from observing already-trained runs; the authors themselves call them associations. With one seed per policy and cell, training variability cannot be separated from treatment effects. The selective-exposure intervention in Appendix E.1 covers only six pairs and yields a modest change (0.719 to 0.661, with four pairs decreasing and two unchanged). These results are therefore better read as descriptive findings on this record than as general causal conclusions.

  2. The measurement chain relies on a single automatic judge and a single caption parser, and the error is not quantified at the concept level. Binding, crossed-pair, and retrieval judgments all depend on one prompt to Qwen3-VL-8B-Instruct. The caption-annotation audit in Appendix F uses only 100 captions, uses another LLM as the reference, and reports pair recall of 56.7% under the strict criterion; the authors explicitly state that "the error in any one concept's exposure count remains unquantified." Since exposure is the central independent variable of the paper, this measurement error propagates into every exposure–outcome conclusion, and the paper does not provide a systematic cross-judge, cross-parser, or human-review robustness analysis. This weakness is consequential because it limits how far the headline dynamics can be trusted beyond this particular record.

Questions for authors
  1. Can you provide at least one independent training seed, even for a single cell such as 10M-M or 100M-S, to estimate training-run variability and show whether the main exposure conclusions in Sections 5.2–5.4 preserve their direction and magnitude across seeds?

  2. The selective-exposure intervention in Appendix E.1 covers only six pairs and leaves two unchanged. Can you report intervention results on a larger set or on higher-exposure pairs, to distinguish an exposure association from an exposure cause?

  3. Caption-annotation pair recall is 56.7% under the strict criterion, and concept-level exposure error is not quantified. Can you provide a frequency-stratified estimate of exposure error, or recompute the key results of Sections 5.2–5.4 with a second parser or a human-labeled subset?

  4. Binding and retrieval judgments depend on a single Qwen3-VL judge. Can you report agreement across judges (different families, or a human-reviewed subset) and state whether the signs and magnitudes in Sections 5.3–5.7 are robust?

2 comments

Thank you for your response. I appreciate the clarification and find it reasonable. However, with respect, I would like to maintain my original evaluation.

Thank you for your reply and for your time. We respect your evaluation.

For completeness, here are the four results we promised. All are complete and in the revision, and none changes a conclusion of the paper.


Q1. Independent seed. Sections 5.2–5.4 keep their direction and size in a second 10M-M run with a new initialization and data order. The exposure–peak correlation of Section 5.4 is r = 0.604 in the new run and 0.605 in the original run (Appendix G).

Pair presentations01–910–99100–999≥1k
Final pass rate, original run0.2390.2960.4460.5840.743
Final pass rate, new run0.2550.2940.4400.5590.725
Pairs reaching onset, original run (%)33.141.961.079.392.0
Pairs reaching onset, new run (%)34.244.361.977.994.6

Q2. Larger intervention. Cutting the exposure of 32 crossed pairs to as little as 0.4% did not lower their crossed peak. The relationship in Section 5.4 is therefore an association, as the paper reads it (Appendix H.4).

Presentations keptPairsMean change in early crossed peakExpected change if exposure set the peak
50%8+0.042−0.09
10%8+0.035−0.30
2%8−0.046−0.51
0.4%8+0.001−0.72
All treated pairs32+0.008
Prompts with no treated pair19 groups−0.0130

Q3. A third parser. Sections 5.2–5.3 also hold under DiscoSG (Appendix H.3).

Pair presentations01–910–99100–999≥1k
Pairs reaching onset, T5 (%)2.87.79.720.939.1
Pairs reaching onset, lexical (%)5.511.113.620.128.9
Pairs reaching onset, DiscoSG (%)3.07.914.916.039.0

Q4. Second judge on 10M-M. Sections 5.2–5.3 also hold under Gemma 4 26B-A4B. The two judges agree on 80.3% of images (Cohen's κ = 0.46), and Gemma is the stricter of the two (Appendix H.2).

Pair presentations01–910–99100–999≥1k
Final pass rate, Qwen0.2390.2960.4460.5840.743
Final pass rate, Gemma0.1350.1710.2850.4140.580
Pairs reaching onset, Qwen (%)44.954.374.986.497.6
Pairs reaching onset, Gemma (%)21.127.045.766.586.6

These checks exist because of your questions. Thank you again for the careful review.

Reviewer_W5w1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Blue Mage, a training record for web-scale text-to-image flow models. It links saved model states to their training history and studies when attribute–entity pairs are learned and when repeated training images can be reproduced.

Critical strengths
  1. Practical value for the community. The paper builds detailed, reproducible training records that help researchers investigate the influence of training data on the behavior of text-to-image models.
  2. The presentation is clear and easy to follow.
Critical weaknesses
  1. Section 3.2 reports only about 57% recall for attribute–entity pairs, which may reduce confidence in the findings about exposure and learning.
Questions for authors
  1. Could the authors verify whether the findings about exposure still hold with more accurate annotations of attribute–entity pairs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.