- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PolyGen, a fully synthetic vision-language pretraining framework designed to reduce generator-specific biases in synthetic training data. The framework combines multiple image generators with complementary fidelity and diversity characteristics, multi-positive contrastive learning, and programmatically generated hard negatives introduced through a curriculum.
The proposed Generator-Invariance Hypothesis suggests that training across heterogeneous generators encourages representations to capture semantic information shared across different synthetic renderings rather than generator-specific artifacts. The authors evaluate this hypothesis through controlled experiments varying generator diversity, positive-sample density, and image-to-image invariance regularization.
Using a fixed 500K-image training budget, PolyGen reports a 19.0% relative improvement in its aggregate multi-task metric over the single-generator, single-positive baseline. Additional experiments show improvements in image-text retrieval and compositional reasoning, with preliminary scaling results at 3M and 12M images.
Critical strengths
-
A systematic investigation of generator heterogeneity under controlled data budgets. The paper addresses an important limitation of synthetic vision-language training: representations may capture generator-specific characteristics rather than transferable visual semantics. Its factorial evaluation of generator heterogeneity and positive-sample density is particularly valuable because it separates improvements from additional synthetic image diversity and repeated observations of the same caption. The seeded comparisons further strengthen the empirical analysis by identifying both performance gains and trade-offs.
-
A practical integration of synthetic data generation and compositional supervision. PolyGen combines heterogeneous generators with structured hard-negative generation and curriculum-based contrastive learning in a coherent training framework. The evaluation covers vision-only classification, cross-modal retrieval, zero-shot recognition, and compositional reasoning. The experiments also demonstrate that the timing of hard-negative introduction matters, providing useful empirical guidance for designing fully synthetic vision-language training pipelines.
Critical weaknesses
-
The incremental contribution of generator heterogeneity is smaller than the headline improvement suggests. The reported 19.0% aggregate improvement compares a configuration with four generators and four positives per caption against a single-generator, single-positive baseline. However, Table 1 shows that increasing positive density to four using only one generator already achieves a 14.1% improvement. The additional benefit attributable to generator heterogeneity is therefore substantially smaller than the overall gain. Furthermore, the seeded comparisons indicate that combining high positive density with generator heterogeneity reduces linear-probe and few-shot performance relative to the corresponding single-generator configuration. These findings do not invalidate the method, but they suggest that the proposed generator-invariance principle yields task-dependent benefits rather than a uniformly better representation.
-
The evidence for large-scale generalization and superiority over real-data training remains limited. The primary experiments use a relatively small 500K-image budget, while the 3M and 12M experiments are limited to single-seed ImageNet zero-shot evaluation. The reported improvements at these larger scales are only 0.25 and 0.64 accuracy points, respectively, making it difficult to establish whether the benefit remains substantial when synthetic training data is scaled. Additionally, the comparison with real CC3M data is affected by differences in caption style and information density, which the authors acknowledge. Since the synthetic captions are more descriptive than the real-data captions, improvements in retrieval cannot be attributed exclusively to the proposed generator ensemble.
Questions for authors
-
In the 3M and 12M experiments, how does PolyGen compare with a single-generator baseline using the same number of positives per caption? The current comparison changes both generator diversity and positive density, making it difficult to determine whether the heterogeneous ensemble remains beneficial at larger scales independently of multi-positive learning.
-
Could the authors clarify how much of the improvement over the real CC3M baseline is attributable to caption distribution differences? For example, would the retrieval advantage persist if the real-data baseline used similarly detailed or LLM-rewritten captions under the same training budget?
4 comments
Part 2
Why 1M and not 3M. The SD1.5 ×4 arm at 1M needed 750k additional SD1.5 renders, which we could generate within the discussion period. A matched arm at 3M would need over two million additional renders before multi-day training could start, which is beyond our compute budget and the time available.
W2: the 3M and 12M results. These runs are single-seed and zero-shot only, and the margin at 3M is about the size of the per-cell seed std we measure at 500k (up to 0.23 top-5, Tab 12). Section 4.4, Appendix J and the Tab 10 caption will present them as preliminary single-seed zero-shot observations, will qualify the statement that the effect "does not vanish with increased data volume", and will add the controlled 1M comparison above. The central contrasts rest on the 500k regime, where they are seeded: the Tab 12 corners and, new in this rebuttal, SANA ×4 and the schedule rows of Tab 2, three seeds each.
W2 / Q2: how much of the CC3M advantage is caption style? We agree that the comparison is confounded by the captions, and we read it as task-dependent parity, not as superiority of synthetic data. We ran the experiment the reviewer suggests: the same 500k CC3M images and protocol, with every alt-text replaced by a text-only LLM rewrite (Qwen3.5-9B), prompted to keep every entity and write one descriptive sentence of 15–20 words, matching our caption length ( words on average; ours ).
| 500k images | captions | LP | FS | Img R@5 | Txt R@5 | ZS top-5 |
|---|---|---|---|---|---|---|
| real CC3M, alt-text | 500k | 50.75 | 59.67 | 11.49 | 14.69 | 13.23 |
| real CC3M, LLM-rewritten (new) | 500k | 50.64 | 59.48 | 10.23 | 11.97 | 10.54 |
| PolyGen | 500k | |||||
| PolyGen | 125k |
(CC3M rows: one run each.) Making the real captions as long and descriptive as ours lowers retrieval ( image, text R@5) and zero-shot ( top-5), and leaves LP/FS unchanged. The retrieval gap to PolyGen therefore widens under matched caption style. This holds also at matched caption count: uses 500k unique captions, like CC3M, and leads the rewritten baseline by / R@5.
We also checked whether our captions resemble the retrieval benchmark captions: 50k captions per set, embedded with a general-purpose sentence encoder (all-mpnet-base-v2), Fréchet distance (FD) between Gaussian fits:
| caption set | words | FD to COCO | FD to Flickr | FD to ours |
|---|---|---|---|---|
| ours | 18.3 | 0.367 | 0.395 | 0 |
| CC3M alt-text | 9.8 | 0.245 | 0.290 | 0.196 |
| CC3M rewritten | 17.1 | 0.232 | 0.281 | 0.151 |
(For scale: COCO Flickr ; split-half noise .) CC3M alt-text is closer to the retrieval benchmarks than our captions, so a caption match to the benchmarks would favour CC3M, not PolyGen; Section 4.2 will replace its statement that our captions are closer to Flickr/COCO with these distances. The rewrite moves CC3M toward our captions and slightly toward the benchmarks, yet its retrieval falls.
A text-only rewrite changes style and length, not content, and without the image it can add unverified details. Prior gains from LLM rewriting use rewrites as augmentation next to the original text [1] or captions that see the image [2], not the full replacement tested here, and Section 4.2 will qualify our citation of them accordingly. We therefore cannot split the CC3M margin exactly into caption and generator effects. What the experiments do isolate is that at fixed captions heterogeneity adds / R@5 (three seeds), and that descriptive captions alone do not give real data that margin. The abstract, introduction and conclusion will present CC3M as a reference point with the "task-dependent parity, caption-confounded" qualifier, and a new appendix will report the rewrite run and the caption distances.
We hope these results address the reviewer's concerns, and we would be glad to discuss any remaining point.
[1] L. Fan, D. Krishnan, P. Isola, D. Katabi, Y. Tian. Improving CLIP Training with Language Rewrites. NeurIPS 2023 (LaCLIP).
[2] Z. Lai, H. Zhang, B. Zhang, et al. VeCLIP: Improving CLIP Training via Visual-enriched Captions. ECCV 2024.
I will keep my original score and look forward to seeing the suggested revisions reflected in the final version of the paper.
We thank the reviewer for the constructive discussion and for the positive assessment of our work. All the revisions described in our reply will be included in the final version of the paper.
Part 1
We thank the reviewer for the careful and constructive review, and for noting that the factorial design separates heterogeneity from positive density: most of this reply relies on exactly that design. Unless marked otherwise, new numbers are three seeds (mean ± sample std), and we write configurations as (m,n+).
W1: the heterogeneity increment and the headline. We agree that the +19.0% of the joint (4,4) configuration is not the heterogeneity effect alone: it combines multiple components, which the paper should separate more clearly. Over three seeds, (4,4) reaches +21.3±1.7% ΔMTL, above the single-run +19.0 of Tab 1. Each step against its own matched reference, from the seeded corners of Tab 12:
The revised abstract, introduction and conclusion will better clarify the contribution of each component (density, heterogeneity, LI2I and the curriculum).
W1: LP/FS cost and task dependence. Over three seeds, heterogeneity costs −1.15 LP and −0.78 FS at n+=4, and nothing measurable at n+=1 (+0.05 / +0.12, inside seed noise). With one image per caption, heterogeneity improves retrieval and zero-shot at no vision-only cost; at high density, it buys larger cross-modal gains (retrieval and zero-shot, the tasks a vision-language model is trained for) for a vision-only cost of about one point. We will revise the introduction and conclusion to state this trade-off explicitly, together with the size of each gain.
Does density make heterogeneity unnecessary? Since density is the larger lever, a natural question is whether a strong single generator at n+=4 matches the pool. We tested this with SANA, the best single generator of Fig 2: we rendered every caption of the n+=4 set three more times with SANA and trained SANA ×4 with three seeds, on the same captions and image budget. SANA ×4 reaches −7.1% ΔMTL, below both the pool (+21.3) and SD1.5 ×4 (+14.9) on every metric (full table in our reply to Reviewer QPkK). So the density gain is not a property of n+ alone: it depends on how much the extra views of a caption differ. Re-seeding one generator adds little image diversity, while n+=4 still means four times fewer captions at a fixed budget. Heterogeneity makes the extra views differ by construction, consistent with it paying off at n+=4 on retrieval and zero-shot. A new appendix table will report SANA ×4 next to the Tab 12 corners.
Q1: the same number of positives per caption at a larger budget. We ran a controlled same-n+ comparison at 1M images, twice the main budget: 250k captions with 4 images each, either four SD1.5 renders per caption (SD1.5 ×4, m=1) or one render from each of the four generators (m=4). Captions, image budget and training recipe are identical between the two arms. One seed per arm:
At matched n+, the retrieval advantage of heterogeneity persists at twice the budget: +1.41 image and +2.21 text R@5, inside the 500k seed range (+1.28 to +1.97 and +1.99 to +3.28 across seeds), and the pool is ahead on all three retrieval benchmarks in both directions. The LP cost is as at 500k, while the FS cost is larger and the small zero-shot gain is not visible at 1M (tied). Both arms gain about equally from the larger budget (e.g. +7.2 to +7.5 image R@5). This is one seed at one larger budget, so we read it as evidence that the heterogeneity effect on retrieval is not specific to 500k, and the revised appendix will report it as such.
A likely reason why the margin does not grow with the budget is that the generators need not contribute equally as data scale up. Our pool splits the image budget uniformly across the four generators, but a generator with low sample diversity, such as the distilled SDXL-Turbo, may saturate earlier: once the model has seen its typical renderings, more of its images add little. A uniform split is therefore not necessarily the best way to build a heterogeneous dataset at larger scale, and choosing per-generator shares (e.g. by diversity) is an interesting direction for future work, which the revised conclusion will mention.