acceptodds
Under review as a conference paper at ICLR 2027

PolyGen: Fully Synthetic Vision-Language Training via Multi-Generator Ensembles

Abstract

Synthetic data offers a scalable solution for vision-language pre-training, yet current state-of-the-art methods typically rely on scaling up a single generative backbone, which introduces generator-specific spectral biases and limits feature diversity. We introduce **PolyGen**, a framework grounded in the *Generator-Invariance Hypothesis*: when images are rendered by an ensemble of architecturally distinct generators, generator-specific artifacts vary across sources while semantic content is shared by construction, and a model trained on the union is encouraged to encode the latter. We treat this hypothesis as a training principle with three testable implications, addressing ensemble composition, invariance regularization, and the timing of hard-negative introduction, each examined through dedicated ablations. PolyGen operationalizes it through a multi-positive strategy exposing the model to disjoint visual manifolds rendered from the same caption, together with a Programmatic Hard Negative curriculum enforcing fine-grained compositional understanding. By reallocating the same image budget from unique captions to multi-source variations, PolyGen yields a more robust feature space, outperforming the leading single-source baseline (SynthCLIP) by on aggregate multi-task benchmarks and on compositional reasoning, and, under an identical protocol, exceeding a real-data (CC3M) baseline on cross-modal retrieval while closing two thirds of the zero-shot gap.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
synthetic data, vision-language models, contrastive learning, CLIP pre-training, generator ensembles, hard negative mining, compositional reasoning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_Gmu5OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PolyGen, a fully synthetic vision-language pretraining framework designed to reduce generator-specific biases in synthetic training data. The framework combines multiple image generators with complementary fidelity and diversity characteristics, multi-positive contrastive learning, and programmatically generated hard negatives introduced through a curriculum.

The proposed Generator-Invariance Hypothesis suggests that training across heterogeneous generators encourages representations to capture semantic information shared across different synthetic renderings rather than generator-specific artifacts. The authors evaluate this hypothesis through controlled experiments varying generator diversity, positive-sample density, and image-to-image invariance regularization.

Using a fixed 500K-image training budget, PolyGen reports a 19.0% relative improvement in its aggregate multi-task metric over the single-generator, single-positive baseline. Additional experiments show improvements in image-text retrieval and compositional reasoning, with preliminary scaling results at 3M and 12M images.

Critical strengths
  1. A systematic investigation of generator heterogeneity under controlled data budgets. The paper addresses an important limitation of synthetic vision-language training: representations may capture generator-specific characteristics rather than transferable visual semantics. Its factorial evaluation of generator heterogeneity and positive-sample density is particularly valuable because it separates improvements from additional synthetic image diversity and repeated observations of the same caption. The seeded comparisons further strengthen the empirical analysis by identifying both performance gains and trade-offs.

  2. A practical integration of synthetic data generation and compositional supervision. PolyGen combines heterogeneous generators with structured hard-negative generation and curriculum-based contrastive learning in a coherent training framework. The evaluation covers vision-only classification, cross-modal retrieval, zero-shot recognition, and compositional reasoning. The experiments also demonstrate that the timing of hard-negative introduction matters, providing useful empirical guidance for designing fully synthetic vision-language training pipelines.

Critical weaknesses
  1. The incremental contribution of generator heterogeneity is smaller than the headline improvement suggests. The reported 19.0% aggregate improvement compares a configuration with four generators and four positives per caption against a single-generator, single-positive baseline. However, Table 1 shows that increasing positive density to four using only one generator already achieves a 14.1% improvement. The additional benefit attributable to generator heterogeneity is therefore substantially smaller than the overall gain. Furthermore, the seeded comparisons indicate that combining high positive density with generator heterogeneity reduces linear-probe and few-shot performance relative to the corresponding single-generator configuration. These findings do not invalidate the method, but they suggest that the proposed generator-invariance principle yields task-dependent benefits rather than a uniformly better representation.

  2. The evidence for large-scale generalization and superiority over real-data training remains limited. The primary experiments use a relatively small 500K-image budget, while the 3M and 12M experiments are limited to single-seed ImageNet zero-shot evaluation. The reported improvements at these larger scales are only 0.25 and 0.64 accuracy points, respectively, making it difficult to establish whether the benefit remains substantial when synthetic training data is scaled. Additionally, the comparison with real CC3M data is affected by differences in caption style and information density, which the authors acknowledge. Since the synthetic captions are more descriptive than the real-data captions, improvements in retrieval cannot be attributed exclusively to the proposed generator ensemble.

Questions for authors
  1. In the 3M and 12M experiments, how does PolyGen compare with a single-generator baseline using the same number of positives per caption? The current comparison changes both generator diversity and positive density, making it difficult to determine whether the heterogeneous ensemble remains beneficial at larger scales independently of multi-positive learning.

  2. Could the authors clarify how much of the improvement over the real CC3M baseline is attributable to caption distribution differences? For example, would the retrieval advantage persist if the real-data baseline used similarly detailed or LLM-rewritten captions under the same training budget?

4 comments

We thank the reviewer for the careful and constructive review, and for noting that the factorial design separates heterogeneity from positive density: most of this reply relies on exactly that design. Unless marked otherwise, new numbers are three seeds (mean ±\pm sample std), and we write configurations as (m,n+)(m, n^+).

W1: the heterogeneity increment and the headline. We agree that the +19.0%+19.0\% of the joint (4,4)(4,4) configuration is not the heterogeneity effect alone: it combines multiple components, which the paper should separate more clearly. Over three seeds, (4,4)(4,4) reaches +21.3±1.7%+21.3 \pm 1.7\% ΔMTL\Delta_{MTL}, above the single-run +19.0+19.0 of Tab 1. Each step against its own matched reference, from the seeded corners of Tab 12:

  • density alone, (1,1)→(1,4)(1,1) \to (1,4): +14.9±0.6%+14.9 \pm 0.6\%;
  • heterogeneity at n+=1n^+=1, (1,1)→(4,1)(1,1) \to (4,1): +7.2±1.1%+7.2 \pm 1.1\%;
  • heterogeneity at n+=4n^+=4, (1,4)→(4,4)(1,4) \to (4,4): +5.0%+5.0\% from the seeded means.

The revised abstract, introduction and conclusion will better clarify the contribution of each component (density, heterogeneity, LI2IL_{I2I} and the curriculum).

W1: LP/FS cost and task dependence. Over three seeds, heterogeneity costs −1.15-1.15 LP and −0.78-0.78 FS at n+=4n^+=4, and nothing measurable at n+=1n^+=1 (+0.05+0.05 / +0.12+0.12, inside seed noise). With one image per caption, heterogeneity improves retrieval and zero-shot at no vision-only cost; at high density, it buys larger cross-modal gains (retrieval and zero-shot, the tasks a vision-language model is trained for) for a vision-only cost of about one point. We will revise the introduction and conclusion to state this trade-off explicitly, together with the size of each gain.

Does density make heterogeneity unnecessary? Since density is the larger lever, a natural question is whether a strong single generator at n+=4n^+=4 matches the pool. We tested this with SANA, the best single generator of Fig 2: we rendered every caption of the n+=4n^+=4 set three more times with SANA and trained SANA ×4 with three seeds, on the same captions and image budget. SANA ×4 reaches −7.1%-7.1\% ΔMTL\Delta_{MTL}, below both the pool (+21.3+21.3) and SD1.5 ×4 (+14.9+14.9) on every metric (full table in our reply to Reviewer QPkK). So the density gain is not a property of n+n^+ alone: it depends on how much the extra views of a caption differ. Re-seeding one generator adds little image diversity, while n+=4n^+=4 still means four times fewer captions at a fixed budget. Heterogeneity makes the extra views differ by construction, consistent with it paying off at n+=4n^+=4 on retrieval and zero-shot. A new appendix table will report SANA ×4 next to the Tab 12 corners.

Q1: the same number of positives per caption at a larger budget. We ran a controlled same-n+n^+ comparison at 1M images, twice the main budget: 250k captions with 4 images each, either four SD1.5 renders per caption (SD1.5 ×4, m=1m=1) or one render from each of the four generators (m=4m=4). Captions, image budget and training recipe are identical between the two arms. One seed per arm:

1M images, n+=4n^+=4LPFSImg Ret.Txt Ret.0-Shot
SD1.5 ×4 (m=1m=1)53.8064.4121.9630.1015.22
four generators (m=4m=4)52.9962.8523.3832.3115.14
difference at 1M−0.80-0.80−1.56-1.56+1.41+1.41+2.21+2.21−0.07-0.07
difference at 500k (3 seeds)−1.15±0.20-1.15 \pm 0.20−0.78±0.15-0.78 \pm 0.15+1.70±0.37+1.70 \pm 0.37+2.44±0.73+2.44 \pm 0.73+0.42±0.17+0.42 \pm 0.17

At matched n+n^+, the retrieval advantage of heterogeneity persists at twice the budget: +1.41+1.41 image and +2.21+2.21 text R@5, inside the 500k seed range (+1.28+1.28 to +1.97+1.97 and +1.99+1.99 to +3.28+3.28 across seeds), and the pool is ahead on all three retrieval benchmarks in both directions. The LP cost is as at 500k, while the FS cost is larger and the small zero-shot gain is not visible at 1M (tied). Both arms gain about equally from the larger budget (e.g. +7.2+7.2 to +7.5+7.5 image R@5). This is one seed at one larger budget, so we read it as evidence that the heterogeneity effect on retrieval is not specific to 500k, and the revised appendix will report it as such.

A likely reason why the margin does not grow with the budget is that the generators need not contribute equally as data scale up. Our pool splits the image budget uniformly across the four generators, but a generator with low sample diversity, such as the distilled SDXL-Turbo, may saturate earlier: once the model has seen its typical renderings, more of its images add little. A uniform split is therefore not necessarily the best way to build a heterogeneous dataset at larger scale, and choosing per-generator shares (e.g. by diversity) is an interesting direction for future work, which the revised conclusion will mention.

Why 1M and not 3M. The SD1.5 ×4 arm at 1M needed 750k additional SD1.5 renders, which we could generate within the discussion period. A matched arm at 3M would need over two million additional renders before multi-day training could start, which is beyond our compute budget and the time available.

W2: the 3M and 12M results. These runs are single-seed and zero-shot only, and the +0.25+0.25 margin at 3M is about the size of the per-cell seed std we measure at 500k (up to 0.23 top-5, Tab 12). Section 4.4, Appendix J and the Tab 10 caption will present them as preliminary single-seed zero-shot observations, will qualify the statement that the effect "does not vanish with increased data volume", and will add the controlled 1M comparison above. The central contrasts rest on the 500k regime, where they are seeded: the Tab 12 corners and, new in this rebuttal, SANA ×4 and the schedule rows of Tab 2, three seeds each.

W2 / Q2: how much of the CC3M advantage is caption style? We agree that the comparison is confounded by the captions, and we read it as task-dependent parity, not as superiority of synthetic data. We ran the experiment the reviewer suggests: the same 500k CC3M images and protocol, with every alt-text replaced by a text-only LLM rewrite (Qwen3.5-9B), prompted to keep every entity and write one descriptive sentence of 15–20 words, matching our caption length (9.8→17.19.8 \to 17.1 words on average; ours 18.318.3).

500k imagescaptionsLPFSImg R@5Txt R@5ZS top-5
real CC3M, alt-text500k50.7559.6711.4914.6913.23
real CC3M, LLM-rewritten (new)500k50.6459.4810.2311.9710.54
PolyGen (4,1)(4,1)500k49.98±0.2249.98 \pm 0.2258.90±0.2058.90 \pm 0.2012.76±0.2712.76 \pm 0.2717.74±0.5217.74 \pm 0.528.90±0.048.90 \pm 0.04
PolyGen (4,4)(4,4)125k50.03±0.1450.03 \pm 0.1458.88±0.1658.88 \pm 0.1616.19±0.2216.19 \pm 0.2222.06±1.0522.06 \pm 1.059.97±0.169.97 \pm 0.16

(CC3M rows: one run each.) Making the real captions as long and descriptive as ours lowers retrieval (−1.26-1.26 image, −2.72-2.72 text R@5) and zero-shot (−2.69-2.69 top-5), and leaves LP/FS unchanged. The retrieval gap to PolyGen therefore widens under matched caption style. This holds also at matched caption count: (4,1)(4,1) uses 500k unique captions, like CC3M, and leads the rewritten baseline by +2.53+2.53 / +5.77+5.77 R@5.

We also checked whether our captions resemble the retrieval benchmark captions: 50k captions per set, embedded with a general-purpose sentence encoder (all-mpnet-base-v2), Fréchet distance (FD) between Gaussian fits:

caption setwordsFD to COCOFD to FlickrFD to ours
ours18.30.3670.3950
CC3M alt-text9.80.2450.2900.196
CC3M rewritten17.10.2320.2810.151

(For scale: COCO ↔\leftrightarrow Flickr =0.219= 0.219; split-half noise ≤0.018\leq 0.018.) CC3M alt-text is closer to the retrieval benchmarks than our captions, so a caption match to the benchmarks would favour CC3M, not PolyGen; Section 4.2 will replace its statement that our captions are closer to Flickr/COCO with these distances. The rewrite moves CC3M toward our captions and slightly toward the benchmarks, yet its retrieval falls.

A text-only rewrite changes style and length, not content, and without the image it can add unverified details. Prior gains from LLM rewriting use rewrites as augmentation next to the original text [1] or captions that see the image [2], not the full replacement tested here, and Section 4.2 will qualify our citation of them accordingly. We therefore cannot split the CC3M margin exactly into caption and generator effects. What the experiments do isolate is that at fixed captions heterogeneity adds +1.70+1.70 / +2.44+2.44 R@5 (three seeds), and that descriptive captions alone do not give real data that margin. The abstract, introduction and conclusion will present CC3M as a reference point with the "task-dependent parity, caption-confounded" qualifier, and a new appendix will report the rewrite run and the caption distances.

We hope these results address the reviewer's concerns, and we would be glad to discuss any remaining point.

[1] L. Fan, D. Krishnan, P. Isola, D. Katabi, Y. Tian. Improving CLIP Training with Language Rewrites. NeurIPS 2023 (LaCLIP).
[2] Z. Lai, H. Zhang, B. Zhang, et al. VeCLIP: Improving CLIP Training via Visual-enriched Captions. ECCV 2024.

I will keep my original score and look forward to seeing the suggested revisions reflected in the final version of the paper.

We thank the reviewer for the constructive discussion and for the positive assessment of our work. All the revisions described in our reply will be included in the final version of the paper.

Reviewer_PvuSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PolyGen studies fully synthetic vision-language pretraining using multiple image generators. It combines heterogeneous generators, multiple positive images per caption, optional image-to-image invariance regularization, and a curriculum of synthetic hard negatives. Experiments examine retrieval, classification, compositional reasoning, and preliminary scaling.

Critical strengths

The fixed-image-budget design and factorial comparison of generator heterogeneity and positive density are valuable. The three-seed corner experiments support a measurable contribution from heterogeneity. The synthetic TripletCLIP-style baseline and matched-image-budget text-only-negative control also help isolate the contribution of the complete framework. The paper reports meaningful trade-offs rather than uniform improvements, and the generator-identification analysis usefully distinguishes ensembling from explicit invariance regularization.

Critical weaknesses
  • The strongest evidence concerns the 500k-image regime. At 3M and 12M images, the reported zero-shot gains are only 0.25 and 0.64 points, with no uncertainty estimates or broader downstream evaluation. These exploratory results leave the practical benefit at larger scales uncertain.

  • The headline gains combine several design choices. The +19.0% aggregate improvement is relative to the single-generator, single-positive baseline; the same-density heterogeneity gain is considerably smaller. Furthermore, the strongest general-transfer and compositional results use different configurations. The contribution of ensembling should therefore be distinguished clearly from positive density, regularization, and schedule selection.

  • The curriculum comparison changes both timing and cumulative hard-negative exposure. Consequently, the schedule family does not fully isolate the claim that timing itself determines compositional performance. Similarly, the mechanism evidence specifically supports the optional invariance loss: generator-identification accuracy increases under ensembling alone and decreases when that loss is enabled, alongside a substantial linear-probe cost.

Questions for authors
  1. What uncertainty accompanies the 3M/12M scaling results? Repeated runs at one larger budget would help establish whether the remaining margin is reliable.

  2. Can schedules with matched cumulative hard-negative exposure distinguish the effect of timing from the effect of negative quantity?

  3. Could the main results explicitly separate the gains from heterogeneity, positive density, invariance regularization, and curriculum, and identify the configuration behind each headline claim?

4 comments

We thank the reviewer for the careful review, and for recognising the value of the fixed-image-budget design and of the factorial comparison of heterogeneity and positive density, with the three-seed corners supporting a measurable contribution from heterogeneity. We also thank the reviewer for raising points that we agree are relevant to clarify and analyse further. We do so below, with new experiments and with the corresponding changes to the revision. New numbers below are three seeds (mean ±\pm sample std) unless marked single-seed.

W1 / Q1: uncertainty at 3M and 12M. The 3M and 12M runs are a supplementary experiment (Appendix J, Tab 10), which the paper already presents as preliminary: limited to ImageNet zero-shot, and read as evidence of persistence beyond 500k rather than as a characterisation of scaling (Appendix J, "Caveats and scope"). Adding uncertainty estimates to them is unfeasible with our compute budget during the rebuttal period: one 3M run takes several days of training on the hardware available to us, and a 12M run several times that.

The revision will make this even clearer in the Tab 10 caption, Appendix J and Section 4.4: one run per arm and per scale, zero-shot only, hence no estimate of seed variance. We will also rephrase "the multi-generator effect … does not vanish with increased data volume" (Appendix J) and "preliminary larger-scale results indicate the effect persists beyond this regime" (Conclusion) as single-run zero-shot observations. The paper's main claims rest on the 500k regime, where each comparison that matters is now seeded: the four m×n+m \times n^+ corners (Tab 12), the best single generator rendered four times against m=4,n+=4m=4, n^+=4 (our reply to Reviewer QPkK), and the two hard-negative schedules of Tab 2 (below).

W2 / Q3: separating the gains, and the configuration behind each headline. We agree that the headline numbers combine multiple design choices and come from different configurations: the revision will say so wherever they appear, labelling each with the configuration it comes from:

headline claimsourcem,n+m, n^+LI2IL_{I2I}hard negativesseeds
+19.0%+19.0\% ΔMTL\Delta_{MTL} over SynthCLIPTab 14, 4nono1 (Tab 12, 3 seeds: +21.3±1.7+21.3 \pm 1.7)
retrieval above CC3M-500k, +4.6+4.6 / +6.7+6.7 R@5Tab 14, 4nono1
62.7% of the zero-shot gap to CC3M closed (33.6% without LI2IL_{I2I})Tab 14, 4yesno1
+11.3%+11.3\% ΔHN\Delta_{HN} (three-seed mean +7.8±3.5+7.8 \pm 3.5)Tab 2 ‡2, 2yeslinear ramp to p=0.25p=0.253

Each contribution separately, from the seeded corners of Tab 12 (three seeds). Absolute points come first; ΔMTL\Delta_{MTL} is each step against its own matched reference:

  • Positive density (m=1m=1, n+n^+ 1→4): +1.57+1.57 zero-shot top-5, +3.01+3.01 / +3.90+3.90 image/text R@5, +1.25+1.25 LP, +0.88+0.88 FS; +14.9±0.6%+14.9 \pm 0.6\% ΔMTL\Delta_{MTL}.
  • Heterogeneity at n+=1n^+=1 (mm 1→4): +0.92+0.92 zero-shot, +1.28+1.28 / +2.02+2.02 R@5, no LP/FS cost (+0.05+0.05 / +0.12+0.12); +7.2±1.1%+7.2 \pm 1.1\%.
  • Heterogeneity at n+=4n^+=4: +0.42+0.42 zero-shot, +1.70+1.70 / +2.44+2.44 R@5, at −1.15-1.15 LP / −0.78-0.78 FS; +5.0%+5.0\%. Heterogeneity therefore helps at both densities: its gain is not absorbed once density is added, and only the two together reach +21.3±1.7+21.3 \pm 1.7; +19.0+19.0 is this joint configuration.
  • LI2IL_{I2I} (Tab 1, single seed, m=4,n+=4m=4, n^+=4 with vs without): +1.48+1.48 zero-shot, −6.0-6.0 LP, −5.6-5.6 FS, −0.4-0.4 / −0.8-0.8 R@5; ΔMTL\Delta_{MTL} falls from +19.0+19.0 to +16.6+16.6. It is a zero-shot operating point, not a general improvement.
  • Hard negatives and curriculum (m=2,n+=2m=2, n^+=2 with LI2IL_{I2I}): they do not move general transfer (ΔMTL\Delta_{MTL} +12.3+12.3 without hard negatives, Tab 1, vs +12.5+12.5 with the full recipe, Tab 2); the contrasts we can measure are on compositional benchmarks. Within Tab 2, the scheduler adds +0.6+0.6 SugarCrepe / +1.3+1.3 SC++ over the unscheduled LHNL_{HN} row (single seed). Lowering the ramp endpoint from 0.5 to 0.25 adds +1.60+1.60 / +1.51+1.51 (three seeds). ΔHN\Delta_{HN} is measured against the m=1,n+=1m=1, n^+=1 row of Tab 2, so it also includes the multi-positive and LI2IL_{I2I} contributions. It is not the curriculum's effect alone, and the revised Tab 2 caption will say so.

W3 / Q2: timing vs cumulative exposure. Following the reviewer's correct observation that Tab 13 changes timing and dose together, we ran the proposed control: three schedules with the same cumulative dose as the conservative ramp (Σp≈4.9\Sigma p \approx 4.9 epoch-units, matched within 1%: 4.84–4.88) but different timing. They use m=2,n+=2m=2, n^+=2, the full recipe and otherwise identical settings:

  • constant p=0.125p = 0.125 from the second epoch on;
  • an early step, p≈0.48p \approx 0.48 for the first 10 epochs, then 00;
  • a late step, p=0p = 0 until the last 10 epochs, then p≈0.48p \approx 0.48.

All rows are three seeds (mean ±\pm sample std).

scheduleΣp\Sigma pSCSC++ ITTZS top-5Img R@5Txt R@5LPFS
linear ramp 0→0.250 \to 0.25 (‡)4.8857.29±1.7057.29 \pm 1.7041.20±1.4041.20 \pm 1.4011.19±0.0911.19 \pm 0.0914.71±0.5014.71 \pm 0.5019.46±0.8219.46 \pm 0.8242.77±0.3442.77 \pm 0.3452.30±0.3052.30 \pm 0.30
constant 0.1250.1254.8856.20±0.5056.20 \pm 0.5039.66±0.4639.66 \pm 0.4611.34±0.1911.34 \pm 0.1914.51±0.2414.51 \pm 0.2419.34±0.7519.34 \pm 0.7542.56±0.1442.56 \pm 0.1452.36±0.2252.36 \pm 0.22
early step4.8456.60±0.3356.60 \pm 0.3339.75±0.2839.75 \pm 0.2811.29±0.1111.29 \pm 0.1114.30±0.0314.30 \pm 0.0319.75±0.3319.75 \pm 0.3342.49±0.3742.49 \pm 0.3752.62±0.3352.62 \pm 0.33
late step4.8456.38±0.6056.38 \pm 0.6040.35±0.7540.35 \pm 0.7510.92±0.1510.92 \pm 0.1514.24±0.2614.24 \pm 0.2619.11±0.3219.11 \pm 0.3242.45±0.0942.45 \pm 0.0952.28±0.2452.28 \pm 0.24

At matched dose, the gradual ramp has the highest mean of the four schedules on both compositional benchmarks: all three alternative timings sit below it, by 0.70.7–1.11.1 points on SugarCrepe and 0.90.9–1.51.5 on SC++ ITT. These gaps are within the seed std, so we present this as a trend rather than an established effect. On zero-shot, retrieval, LP and FS, the shapes stay within 0.50.5 points of the ramp. The dose contrast, instead, is seeded: the ramp to 0.25 is ahead of the ramp to 0.5 (Σp≈9.8\Sigma p \approx 9.8) on SugarCrepe on each of three paired seeds, and on zero-shot by about twice the seed std (table in our reply to Reviewer QPkK, where we also explain that the 0.25 endpoint was chosen on SugarCrepe / SugarCrepe++).

We will therefore rewrite "compositional performance is not governed by cumulative exposure" and "timing governs the outcome" (Section 4.3, Appendix N, Tab 13 caption) to state that cumulative dose has a measurable effect and that, at matched dose, a gradual ramp has the highest mean among the schedules we tested, a trend within seed variation. The revision will also add the rows above to Tab 13 (Appendix N), rephrase implication (3) in the introduction as a question about the schedule, and remove "each validated by seeded ablations" from the Conclusion for the hard-negative part.

W3: the mechanism. We agree. Tab 11 shows that ensembling alone does not lower the decodability of generator identity: it rises from 81.26% to 88.47%, close to a LAION-2B CLIP at 87.96%. The drop comes from LI2IL_{I2I}: 88.47% to 68.33%, with the probe margin at the random-initialisation level. Appendix L already attributes the invariance mechanism to LI2IL_{I2I} rather than to ensembling, but the abstract, introduction and conclusion still state the hypothesis as if training on the union were sufficient. The revision will reword it consistently in all three places: heterogeneity supplies cross-source variation of the same caption, and LI2IL_{I2I} is what converts it into measurable invariance. Without LI2IL_{I2I}, what ensembling buys is the downstream gains of Tab 1 and Tab 12, not invariance in the probe's sense. The linear-probe cost (48.9 → 42.9 at m=4,n+=4m=4, n^+=4) belongs to that operating point, which trades vision-only probing for zero-shot (9.86 → 11.34). We will present LI2IL_{I2I} as an option rather than the default, since every heterogeneity result in Tab 1 and Tab 12 holds without it.

Presentation. With each headline number labelled by its configuration in the abstract and introduction, and the contribution of each component clarified in Section 4, the paper should be easier to follow.

We hope these answers address the reviewer's concerns, and if they do, we would be grateful if the reviewer would take them into account in their assessment. We would be glad to discuss any remaining point.

Thank you for the additional experiments and clarifications. The remaining limitation is the evidence at larger scales: the 3M/12M results remain single run observations, and the new 1M comparison provides limited additional support.

Please incorporate the proposed corrections consistently into the revised manuscript. With these clarifications, I maintain my weak accept recommendation.

We thank the reviewer for engaging with our responses and for maintaining the weak accept recommendation. We agree that the 3M and 12M runs remain single-run observations. As stated in Appendix J, they are a supplementary experiment, presented as a preliminary, zero-shot-only indication beyond 500k, and none of the main claims of the paper rests on them: those rest on the 500k regime, where every central comparison is now seeded. The revision will incorporate all the proposed corrections consistently, and Section 4.4, Appendix J, the Tab 10 caption and the Conclusion will state explicitly the preliminary, single-run scope of the 3M/12M results and of the new 1M comparison.

Reviewer_QPkKOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Fully synthetic CLIP pre-training. Instead of one text-to-image model, each LLM-written caption is rendered by four (SD 1.5, SD 2, SANA, SDXL-Turbo) and all four are used as positives, with counterfactual captions ramped in as hard negatives. At a fixed 500k image budget this beats a single-generator SynthCLIP baseline across the board, and a real CC3M-500k baseline on retrieval.

Critical strengths
  1. Tab 1. At a fixed 500k images, spending them on several renderings of fewer captions beats spending them on more unique captions: +19.0% ΔMTL over SynthCLIP, and both retrieval directions above a real CC3M-500k baseline under the same protocol (+4.6 and +6.7 R@5). Beating real data on retrieval with no real data in the pipeline is worth publishing, and people can act on it right away.

  2. Fig 2 and sec 3.2. They adapt Recognizability and Diversity to the open-vocabulary setting and then use them as a real admission test, so a generator earns its role by measured position rather than by name or release date. That procedure outlasts the four models they happened to pick.

Critical weaknesses
  1. Every m=1 row in Tab 1 is SD 1.5, which is their own low-fidelity Diversity Expert. So "ensembles beat single-source scaling" only ever gets tested against the weakest model in the pool, never the best one. Fig 2 has all 15 subsets and tabulates none of them, so single SANA and single SDXL-Turbo show up nowhere I can check. Their own last column already puts heterogeneity at +3.8%, not the +19.0% the abstract leads with. Tabulate Fig 2, then run the best single generator at n+=4 against m=4 with three seeds.

  2. The +11.3% compositional number is the conservative schedule, picked out of the four in Tab 13 on the same SugarCrepe it then gets reported on. The linear ramp the method actually specifies gives +6.1%. Seed those rows, or put the linear-ramp number in the abstract.

Questions for authors

All three are clarifications on existing results, not requests for future work.

  1. What is delta MTL for each generator on its own in Fig 2, specifically SANA alone and SDXL-Turbo alone? If either already matches your m=4, n+=1 row, my reading of the main claim changes.

  2. Was the conservative schedule chosen after looking at SugarCrepe and SC++ accuracy? And are there seed repeats for any row of Tab 2 besides the corner runs?

  3. Can you report ImageNet top-1 next to top-5 for the Tab 12 cells? Top-5 sits at 8 to 10 across that table, and since ΔMTL averages relative gains, I want to know how much of the +19.0% is a floor effect.

2 comments

We thank the reviewer for a precise review and for stating concretely what would change their assessment. Unless marked otherwise, new numbers are three seeds (mean ±\pm sample std).

Q1: each generator alone in Fig 2. Fig 2 comes from a preliminary study on a 500k subset of the SynthCLIP captions (n+=1n^+=1, one seed per subset). We ran it to check whether mixing generators is worth pursuing. As plotted in Fig 2, the single-generator ΔMTL\Delta_{MTL} values, relative to SD1.5 on the same captions, are:

SANASD2SDXL-Turbo
ΔMTL\Delta_{MTL} vs SD1.5+1.13+1.13−0.37-0.37−8.60-8.60

The revised appendix will tabulate all 15 subsets. SANA is the best single generator by ΔMTL\Delta_{MTL}, and SD1.5 is second, ahead of SD2 and SDXL-Turbo. We keep SD1.5 as the m=1m=1 generator for two reasons. It is the generator of SynthCLIP, the baseline we compare against. It is also the best single generator on zero-shot (8.14 vs 6.90 ImageNet top-5 for SANA in this study). This still leaves open the comparison against the strongest single generator, so we ran the test proposed in W1 with SANA.

W1: the best single generator at n+=4n^+=4 vs m=4m=4, three seeds. We rendered every caption of the n+=4n^+=4 caption set three more times with SANA (same sampler settings as the dataset, new seeds). We then trained SANA ×4 (m=1,n+=4m=1, n^+=4) with three seeds, on the same captions and with the same 500k image budget as m=4,n+=4m=4, n^+=4. ΔMTL\Delta_{MTL} is against the seeded m=1,n+=1m=1, n^+=1 baseline.

LPFSImg R@5Txt R@5ZS top-5ZS top-1ΔMTL\Delta_{MTL}
m=1,n+=1m=1, n^+=1 (SD1.5)49.93±0.3249.93 \pm 0.3258.78±0.0858.78 \pm 0.0811.48±0.0911.48 \pm 0.0915.72±0.5315.72 \pm 0.537.98±0.237.98 \pm 0.232.61±0.072.61 \pm 0.07-
SD1.5 ×4 (m=1,n+=4m=1, n^+=4)51.18±0.2251.18 \pm 0.2259.66±0.0359.66 \pm 0.0314.49±0.3114.49 \pm 0.3119.62±0.3319.62 \pm 0.339.55±0.019.55 \pm 0.013.24±0.043.24 \pm 0.04+14.9+14.9
SANA ×4 (m=1,n+=4m=1, n^+=4), new46.73±0.4946.73 \pm 0.4954.70±0.1654.70 \pm 0.1612.03±0.2312.03 \pm 0.2313.91±0.4713.91 \pm 0.476.74±0.176.74 \pm 0.172.02±0.062.02 \pm 0.06−7.1-7.1
m=4,n+=4m=4, n^+=450.03±0.1450.03 \pm 0.1458.88±0.1658.88 \pm 0.1616.19±0.2216.19 \pm 0.2222.06±1.0522.06 \pm 1.059.97±0.169.97 \pm 0.163.42±0.073.42 \pm 0.07+21.3+21.3

The m=4,n+=4m=4, n^+=4 row is the configuration of Tab 1. Averaged over three seeds (the seeds of Tab 12), its ΔMTL\Delta_{MTL} is +21.3±1.7+21.3 \pm 1.7, higher than the single-run +19.0+19.0 printed in Tab 1: the seeded estimate confirms, and slightly exceeds, the gain reported in the paper.

m=4,n+=4m=4, n^+=4 beats SANA ×4 on every metric: +3.30+3.30 LP, +4.18+4.18 FS, +4.16+4.16 image and +8.15+8.15 text R@5, and +3.23+3.23 / +1.40+1.40 zero-shot top-5 / top-1. SANA is the best single generator of Fig 2, so this is the comparison specified in W1, and the pool wins it on all metrics. The strongest single-generator configuration at n+=4n^+=4 in our runs is SD1.5 ×4, and against it the pool wins on retrieval and zero-shot but not on LP/FS.

SANA ×4 is also well below SD1.5 ×4. Re-seeding one generator adds little image diversity, because its renders of a caption tend to share the same composition, and at a matched image budget n+=4n^+=4 also means 4× fewer captions. Extra views of one generator therefore pay off only as far as its renders differ: SD1.5, the highest-Diversity generator in Fig 2, gains +14.9+14.9 from them, while for SANA, the lowest-Diversity one, they do not make up for the lost caption diversity. Compared with SD1.5 ×4, m=4,n+=4m=4, n^+=4 is ahead on image R@5 (+1.70+1.70), text R@5 (+2.44+2.44) and zero-shot (+0.42+0.42 top-5), and behind on LP (−1.15-1.15) and FS (−0.78-0.78). This is the trade-off reported in Tab 12.

W1: the size of the effect. We agree that the gain of the joint m=4,n+=4m=4, n^+=4 configuration is not the heterogeneity effect alone: it combines several components, of which heterogeneity is one, which the paper should separate more clearly. From the seeded means, each against its matched reference and writing configurations as (m,n+)(m, n^+), heterogeneity gives +5.0%+5.0\% ΔMTL\Delta_{MTL} at n+=4n^+=4 ((4,4)((4,4) vs (1,4)(1,4), relative to (1,4)(1,4), hence not the difference between the two rows' ΔMTL)\Delta_{MTL}) and +7.2±1.1%+7.2 \pm 1.1\% at n+=1n^+=1 ((4,1)((4,1) vs (1,1))(1,1)). Density alone gives +14.9±0.6%+14.9 \pm 0.6\% ((1,4)((1,4) vs (1,1))(1,1)). That these components add up is not obvious a priori, since extra views of a single generator do not reliably help: they take SANA to −7.1-7.1 but SD1.5 to +14.9+14.9. Heterogeneity instead helps at both densities, so its gain is not absorbed once density is added, and only the two together reach +21.3±1.7+21.3 \pm 1.7. The revised abstract, introduction and conclusion will better clarify the contribution of each component (density, heterogeneity, LI2IL_{I2I} and the curriculum).

W2 / Q2: schedule selection and seeds. The conservative schedule is the method's linear ramp, only with a lower endpoint. Both schedules increase pp linearly over the 40 epochs, p(e)=(e/40)⋅pmaxp(e) = (e/40) \cdot p_{max}, with pmax=0.25p_{max} = 0.25 (‡) or pmax=0.5p_{max} = 0.5 (Tab 2 row 6). pmaxp_{max} is a hyperparameter of the method. Second, the answer to Q2 is yes. We tried the schedules of Tab 13 and reported the best one, and we made that choice on SugarCrepe / SugarCrepe++, the same benchmarks the number is reported on. Before this rebuttal, no Tab 2 row other than the corners had seed repeats. We have now run both schedule rows (row 6 and ‡) with three seeds each:

scheduleLPFSImg R@5Txt R@5ZS top-5SCSC++ ITTΔHN\Delta_{HN}
linear, pmax=0.5p_{max}=0.5 (row 6)42.34±0.4542.34 \pm 0.4552.29±0.0752.29 \pm 0.0714.27±0.3214.27 \pm 0.3219.52±0.9619.52 \pm 0.9610.95±0.1410.95 \pm 0.1455.69±0.8055.69 \pm 0.8039.69±0.6839.69 \pm 0.68+4.3+4.3
linear, pmax=0.25p_{max}=0.25 (‡)42.77±0.3442.77 \pm 0.3452.30±0.3052.30 \pm 0.3014.71±0.5014.71 \pm 0.5019.46±0.8219.46 \pm 0.8211.19±0.0911.19 \pm 0.0957.29±1.7057.29 \pm 1.7041.20±1.4041.20 \pm 1.40+7.8+7.8

As in the paper, ΔHN\Delta_{HN} is computed against Tab 2 row 1. Seed 1 of each row was re-evaluated with the same pipeline as the new seeds, which moves it by at most 0.11 accuracy points from the printed values. The lower endpoint is ahead on every seed, by +5.4+5.4, +1.6+1.6 and +3.6+3.6 ΔHN\Delta_{HN} in paired comparisons (SC +2.81+2.81 / +0.93+0.93 / +1.04+1.04), so the ordering holds. However, the +11.3%+11.3\% we reported is the best of the three seeds. The revised abstract will report the seed mean, "+7.8% on average over three seeds (up to +11.3%)". Tab 2 will give mean ±\pm std for both rows, and its caption will state that pmaxp_{max} was selected on these benchmarks. Since the two rows differ only in pmaxp_{max}, i.e. in cumulative hard-negative exposure, Section 4.3 and Appendix N will state that this dose has a measurable, seeded effect. On general transfer the two endpoints are close. The lower one is ahead on zero-shot (+0.25+0.25 top-5, about twice the seed std), image R@5 and LP are within seed noise, and FS and text R@5 are tied.

Q3: top-1 and the floor effect. Top-1 for the Tab 12 cells is (1,1)(1,1) 2.61±0.072.61 \pm 0.07, (4,1)(4,1) 3.01±0.073.01 \pm 0.07, (1,4)(1,4) 3.24±0.043.24 \pm 0.04 and (4,4)(4,4) 3.42±0.073.42 \pm 0.07; (4,4)(4,4) + LI2IL_{I2I} reaches 3.963.96 (one seed). In absolute terms, (4,4)(4,4) over (1,1)(1,1) is +1.99+1.99 top-5 and +0.8+0.8 top-1 points, +4.71+4.71 image and +6.34+6.34 text R@5, and +0.1+0.1 on LP and FS.

As the reviewer notes, ΔMTL\Delta_{MTL} averages relative gains, so low-base metrics weigh more in it. For the Tab 1 number itself (seed 1), dropping the zero-shot term moves +19.0+19.0 to +18.5+18.5, and using top-1 instead of top-5 moves it to +20.1+20.1. On the seeded means, the same variants take +21.3+21.3 to +20.4+20.4 and +22.6+22.6. So the zero-shot floor accounts for little of the aggregate, and top-1, which is closer to the floor, raises it slightly. Most of the aggregate comes from retrieval (seed 1: +39%+39\% image and +35%+35\% text R@5, relative), which also starts from a low base. Across the 11 Tab 1 configurations with m,n+∈{1,2,4}m, n^+ \in \{1, 2, 4\}, the ranking is unchanged with top-1 (Spearman ρ=0.99\rho = 0.99). Without the zero-shot term, the plain-CLIP configurations keep their order and the LI2IL_{I2I} configurations move down (ρ=0.89\rho = 0.89), consistent with LI2IL_{I2I}'s gain being concentrated in zero-shot.

We hope these results address the reviewer's concerns. In particular, since the pool wins the three-seed comparison proposed in W1, we would be grateful if the reviewer would consider revising their score. We would be glad to discuss any remaining point.

Comments on the paper

1 comment

We thank the reviewers and the Area Chair for their time. During the discussion we ran the following new experiments, all at the paper's fixed 500k image budget and with three seeds unless stated:

  • Best single generator vs the pool (QPkK W1). SANA rendered four times (m=1,n+=4m=1, n^+=4) stays below m=4,n+=4m=4, n^+=4 on every metric (ΔMTL\Delta_{MTL} −7.1-7.1 vs +21.3±1.7+21.3 \pm 1.7).
  • Separation of the gains (QPkK, PvuS, Gmu5). From the seeded corners: density +14.9±0.6%+14.9 \pm 0.6\%, heterogeneity +7.2±1.1%+7.2 \pm 1.1\% at n+=1n^+=1 and +5.0%+5.0\% at n+=4n^+=4. The revision will label every headline number with the configuration it comes from.
  • Hard-negative schedule (QPkK, PvuS). Seeded linear schedules (ΔHN\Delta_{HN} +7.8±3.5+7.8 \pm 3.5 with pmax=0.25p_{max} = 0.25, +4.3±1.6+4.3 \pm 1.6 with pmax=0.5p_{max} = 0.5) and three schedules at matched cumulative exposure.
  • Caption style vs real data (Gmu5). CC3M with LLM-rewritten captions: its retrieval decreases, so the advantage of PolyGen is not explained by caption style.
  • Larger scale (Gmu5, PvuS). A controlled 1M comparison (SD1.5 ×4 vs m=4,n+=4m=4, n^+=4, one seed each), which required 750k new renders.

We are grateful for the constructive exchanges with the reviewers. We did our best to run every experiment that was feasible within the time of the discussion period, and we will carefully incorporate in the revised paper all that was discussed and presented during this process. We thank the Area Chair and the reviewers once more for their time and work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.