acceptodds
Under review as a conference paper at ICLR 2027

T-DIG: Text-Conditioned Diversity in Image Generation

Abstract

Text-to-image (T2I) platforms typically return several images per prompt so users can explore different interpretations and choose one that matches their intent. Most existing methods for diversity optimize pairwise distances in a latent embedding space, which are hard to map to interpretable attributes and don't account for user preferences. Some users may want variations in the background, whereas others may want variations in artistic styles. In this paper, we introduce T-DIG, a diversity framework for image generation and editing. T-DIG is an iterative framework comprising of multiple modules that lets users type prompts, inspect default generations, request for variations using natural language, and continue refining outputs until they're satisfied. At its core, T-DIG uses a LLM policy that produces axis-conditioned prompts or edit instructions for a frozen generator or editor. We train this policy with GRPO using a text-conditioned diversity reward and alignment penalties. Because the renderer remains frozen, T-DIG applies to open-weight, distilled, and proprietary models. We collect human-written axes to build training and validation data, and derive a test set from user interactions with \method, all of which we release as the first dataset for text-conditioned prompt diversity in image generation. T-DIG far exceeds baselines in improving text-conditioned diversity while preserving base-prompt alignment across a variety of metrics and human judgements. Our code and data is here: https://anonymous.4open.science/r/t-dig-paper

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
diversity in image generation, GRPO, reward modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6LfJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

T-DIG formulates image-set diversity as generation conditioned on a user-written axis of variation. Two LoRA-adapted language policies produce prompt sets or editing instructions for frozen image generators/editors. GRPO optimizes a normalized TPIPS-conditioned Vendi reward with baseline-relative alignment and preference penalties and a length penalty; the editing policy also receives a DreamSim-based drift penalty. Training axes are primarily generated by a VLM using human-written pilot responses as examples, while the test requests come from user interactions. The paper reports diversity improvements across four open-weight generators, transfer to two API generators, and human comparisons, with some alignment and preference tradeoffs.

Critical strengths

The task formulation addresses differences in users' intentions by allowing free-form variation axes for the same prompt without requiring a fixed attribute taxonomy. The text-policy interface also makes the approach applicable to frozen renderers from different model families. Table 3 reports diversity gains across four generators, while Table 1 shows that people generally judge the sets produced at α=0.5 as more varied along the requested axis than those from the external baselines. The 50-prompt API transfer experiment provides additional, although limited, evidence of applicability beyond open-weight models. The comparison with the untrained policy makes the contribution of training more credible. This control receives the same axis-conditioned instruction, yet the α=1 adapter adds approximately 0.063–0.114 normalized TPIPS-Vendi across the four generators in Table 3. Table 2 also shows why optimizing diversity alone is insufficient: it raises the diversity score to 0.617 while reducing VQAScore to 0.228. The alignment, length, and preference penalties address different observed failure modes, providing useful evidence for the reward design rather than only a final aggregate result.

Critical weaknesses

Incomplete reporting limits the assessment of the human evidence. Section 4.4 describes 80 participants and 400 test pairs, whereas RQ5 reports 53 participants and 415 pairs without explaining their relationship. Table 1 also omits cell-level sample sizes, win/loss/tie counts, and a precise definition of the sign test underlying its significance statement. This matters particularly for the joint diversity-and-alignment criterion: a set must be more varied and at least as well aligned to count as a joint win, so neither method may satisfy the criterion. Consequently, the reported joint win rates of 47.6%–58.3% against the untrained policy cannot, by themselves, establish or refute statistical significance. The authors should specify the null hypothesis, denominator, handling of ties and cases where neither method jointly wins, and which columns the caption's significance statement covers. Until these details are provided, the strength of the claimed human-evaluation advantage remains uncertain. The evidence is also less complete for preserving required content outside the requested axis and for iterative editing than for increasing diversity. TPIPS-Vendi is both a training objective and the principal automatic diversity metric; the human comparisons are helpful, but do not directly quantify retention of individual prompt constraints. Tables 1–3 do not clearly separate generation from editing or characterize drift over successive rounds, although Appendix A.2 reports unintended hair-color changes and accumulating editing artifacts. At α=1, Table 3 additionally shows HPSv3 decreases of about 6.1% for Qwen-Image and 10.4% for FLUX.2 [klein] relative to i.i.d. sampling. Thus, the current evidence supports a useful diversity/quality tradeoff more strongly than a general preservation claim.

Questions for authors
  1. Please explain how Section 4.4's 80 participants/400 pairs relate to RQ5's 53 participants/415 pairs, including cohorts, rounds, and exclusions. For Table 1, please provide cell-level sample sizes, win/loss/tie counts or the underlying rating differences, confidence intervals, and the precise sign-test definition. Which comparisons does the caption's significance statement cover, and how are cases where neither set jointly wins handled?
  2. Please separate generation and editing sample sizes/results and specify the editing ν and δ and the construction of its baseline-relative thresholds. Under matched axes and budgets, how do trained and untrained editing policies compare on target-axis variation, non-target content preservation, and drift over successive rounds?
  3. What motivated α=0.5 for human evaluation and α=1 for the main automatic comparison? Can you report the automatic diversity, alignment, and preference results at the human-study setting as well, so that the trained-versus-untrained tradeoff can be compared at the same operating point and noise protocol?
4 comments
AuthorsOpenReview ↗

Weakness 1: Participant, pair counts (Section 4.4 vs. RQ5) and Table 1 sample sizes, W/T/L ...

Participant and pair counts (Section 4.4 vs. RQ5): Apologies for the inconsistency! Both places describe the interactive study in which we collect the test set. Section 4.4 states our planned design which was to have 80 participants contributing 5 prompts each, for 400 pairs. However, many participants who enjoyed the study contributed more than 5 prompts, so the final test set has 405 (prompt, axis) pairs from 58 contributors. The original number of submissions was 415 (the number in RQ5), but we filtered out 10 samples whose axes were judged as poor in Stage 2 (L307–309). RQ5 reports the ratings people gave to their outputs in Stage 1, and only 53 of the 58 contributors completed the exit questionnaire; so this is where the 53 / 415 numbers come from. We will update Section 4.4 in the final version!

Table 1 sample sizes, win/loss/tie counts, and sign-test definition

The numbers are correct, but they are computed while accounting for ties, which our caption did not explain. Each comparison has three possible outcomes: T-DIG wins, the baseline wins, or a tie (no majority of raters for either method). Ties are more frequent in the diversity + alignment setting, since a win there requires diversity to be greater and alignment to be greater than or equal to the baseline.

The percentage in Table 1 is T-DIG wins divided by all comparisons, ties included. A tie therefore lowers the reported win rate even though T-DIG did not lose that comparison; for example, 10 wins, 10 ties and 1 loss is reported as 10/21 = 47.6%. The sign test is two-sided and compares wins against losses only. A win rate at or below 50% therefore does not mean T-DIG lost more often than it won. We will replace the win percentages with wins / ties / losses for every cell (full table below) and state these details clearly in the text. Please let us know if we can clarify further.

Each cell is wins / ties / losses for T-DIG (α = 0.5), with the two-sided sign-test p-value.

CriterionModelvs. I.I.D.vs. CADSvs. Group Inf.vs. T-DIG α = 0
More variedQwen-Image23 / 2 / 0 (p<0.001)25 / 1 / 0 (p<0.001)22 / 1 / 2 (p<0.001)18 / 4 / 2 (p<0.001)
More variedSD3.5 Large21 / 2 / 0 (p<0.001)18 / 2 / 3 (p=0.001)18 / 4 / 1 (p<0.001)15 / 1 / 3 (p=0.008)
More variedSD3.5 Medium13 / 0 / 1 (p=0.002)12 / 2 / 0 (p<0.001)9 / 4 / 1 (p=0.021)9 / 5 / 0 (p=0.004)
More variedFLUX.2 [klein]20 / 1 / 0 (p<0.001)19 / 2 / 2 (p<0.001)20 / 2 / 1 (p<0.001)13 / 7 / 1 (p=0.002)
More varied + alignedQwen-Image17 / 8 / 0 (p<0.001)21 / 5 / 0 (p<0.001)19 / 6 / 0 (p<0.001)14 / 10 / 0 (p<0.001)
More varied + alignedSD3.5 Large19 / 4 / 0 (p<0.001)14 / 7 / 2 (p=0.004)13 / 9 / 1 (p=0.002)10 / 7 / 2 (p=0.039)
More varied + alignedSD3.5 Medium8 / 5 / 1 (p=0.039)9 / 5 / 0 (p=0.004)6 / 7 / 1 (p=0.125)7 / 7 / 0 (p=0.016)
More varied + alignedFLUX.2 [klein]13 / 8 / 0 (p<0.001)16 / 6 / 1 (p<0.001)16 / 6 / 1 (p<0.001)10 / 10 / 1 (p=0.012)
AuthorsOpenReview ↗

Weakness 2: Preserving content outside the requested axis

We agree, and we do not claim that T-DIG leaves all non-target content unchanged: our claim is that it raises diversity along the requested axis while preserving base-prompt alignment (L50–51). However, we do briefly discuss this as a motivation to add the length penalty (L200–204). All images in a set also share the same noise latent, so the variation comes only from the prompt. That said, we agree that off-axis change should be quantified, so we measure it for both generation and editing. We also have more discussions on how the strength control which the user sets, controls for this a bit, and some diversity is good because it helps with creative exploration (but more on this below).

Generation: We compute the TPIPS distance of the set conditioned on (i) the requested axis and (ii) other axes related to the prompt. When constructing our training and validation data, we ask an LLM to propose 3–5 plausible axes of variation per prompt; we use the first one for (i) and the rest for (ii). We test this across multiple diversity strengths α (the scale of the trained adapter; α = 0 is the untrained policy) and baselines. We report results for the fastest model in our setup (FLUX.2 [klein]) on 200 validation prompts:

MethodTPIPS diversity gain (requested axis)TPIPS diversity gain (other axes)Pairs with more gain on requested axis
T-DIG α = 00.190.0196.9%
T-DIG α = 0.250.240.0597.4%
T-DIG α = 0.50.280.0996.4%
T-DIG α = 0.750.290.1197.4%
T-DIG α = 10.310.1395.9%
CADS (baseline)0.120.1248.5%
Group Inference (baseline)0.050.0446.9%

The diversity gain is larger on the requested axis than on the other axes for 96–97% of pairs at every strength. The baselines, which do not see the axis, spread their gain evenly across requested and non-requested axes, so they gain more on the requested axis for only 47–49% of pairs, i.e., chance.

Note that off-axis diversity is not expected to be zero. Axes of the same prompt can overlap in meaning: for example, if background is the requested axis and lighting is one of the other axes, changing one naturally changes the other. We also view some off-axis variation as a good thing: our goal is to show users a variety of outputs that help them explore different interpretations of a prompt, and off-axis creativity helps with that. Importantly, off-axis variation increases with diversity strength, which is a user control: at low strength there is almost no off-axis change (0.01 at α = 0), and at high strength there is some, so users can choose based on their preference.

Editing: In editing, the user starts from a set of images for a prompt and asks T-DIG to make them more diverse along one axis. T-DIG writes one edit instruction per image, and the editor applies it. We run the same test as above, but the diversity gain is measured relative to the source images (before editing) instead of an i.i.d. set.

Strength αTPIPS diversity gain (requested axis)TPIPS diversity gain (other axes)Pairs with more gain on requested axis
0 (untrained)0.200.0595.9%
0.250.230.0698.8%
0.50.240.0699.0%
0.750.250.0697.9%
10.250.0797.9%

At every strength, the diversity gain is 3.5–4.2× larger on the requested axis than on the other axes, and it is larger for 96–99% of pairs. Increasing the strength from 0 to 1 adds 0.05 on the requested axis and only 0.02 on the others, so off-axis change in editing is small and nearly flat across strengths.

We will add this analysis to the paper!

AuthorsOpenReview ↗

Weakness 3: Separating generation from editing

We originally omitted the editing results because the findings are similar to the generation results, but for completeness we will add them to the final version of the paper (along with qualitative examples)! The results below are across different diversity strengths:

EditorαTPIPS diversity ↑VQAScore ↑Drift ↓HPSv3 ↑ImageReward ↑
Qwen-Image00.3800.8020.20610.780.89
Qwen-Image0.50.4130.8400.20311.751.14
Qwen-Image10.4230.8460.19611.791.17
SD3.5 Large00.2720.8520.20611.991.20
SD3.5 Large0.50.3090.8540.19312.251.25
SD3.5 Large10.3230.8550.19512.371.28
SD3.5 Medium00.2710.8470.19411.571.17
SD3.5 Medium0.50.3240.8560.19111.911.24
SD3.5 Medium10.3490.8630.20911.991.24
FLUX.2 [klein]00.3630.8090.17811.430.97
FLUX.2 [klein]0.50.4080.8330.17111.911.11
FLUX.2 [klein]10.4200.8400.18012.191.15

Q1 and Q2 are addressed in our responses to the weaknesses above.

Q3: We use α = 0.5 as the default in the human studies because it balances the trade-off between diversity and alignment well. It was simply a design choice because collecting human ratings on multiple alpha values is expensive. The main automatic comparison (Table 3) reports the fully trained policy (α = 1), but since α only scales the trained adapter, the automatic metrics can be computed at any α. The off-axis and editing analyses above already report α = 0.5 alongside α = 0 and α = 1, and we will similarly add automatic diversity, alignment, and preference results at multiple α values (including α = 0.5) for the main results in the final version!

Reviewer_pCt6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces T-DIG, which reformulates diversity in text-to-image generation as a set generation problem controlled by textual conditions. Given a base prompt, a user-specified variation axis, and a set size, the system generates a set of images that vary along the specified axis while preserving the original semantics as much as possible. Rather than updating the image generator, T-DIG uses an LLM policy post-trained with GRPO to generate expanded prompts or, for existing images, editing instructions, which are then rendered by a frozen generator/editor.

Critical strengths
  1. The research problem is well-motivated and practically relevant. Multi-image generation interfaces need more than unconditional diversity. They need variations along specific directions that align with users’ current creative intent.
  2. The task formulation and system integration offer a degree of novelty. Individual components are not new in themselves. However, bringing together free-form textual axes, set-level optimization, generation/editing, and interactive refinement under the goal of user-request-driven, text-conditioned diversity constitutes a valuable and effective first combination.
  3. Convincing Experimental Results. Both the objective metrics and subjective evaluations provide convincing evidence of the effectiveness of the proposed method.
Critical weaknesses
  1. Off-Axis Leakage. When the base prompt leaves attributes such as background, composition, and style unspecified, can the system freely alter these non-target attributes without penalty? Some examples appear to demonstrate greater variation along the requested axis, but this alone is insufficient to support the stronger claim that the method enables precise and disentangled variation along the specified axis.
  2. Limited Comparisons with Existing Prompt Optimization Methods. The main learnable component of T-DIG appears to be axis-conditioned prompt-set rewriting. While the results demonstrate the benefits of GRPO, additional comparisons with a broader range of prompt optimization and enhancement methods would help better establish the relative advantages of the proposed approach.
Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Weakness 1: Off-axis Leakage

Thanks for your review!

We agree that our results do not establish precise, disentangled control, and we'd like to note that we do not make this claim in the paper as well. Our claim is that T-DIG produces sets that differ along the requested axis while following the base prompt (L50–51), i.e., it raises on-axis diversity while preserving base-prompt alignment. We explicitly identify off-axis leakage as a motivation for the length penalty (L200–204). All images in a set also share the same noise latent, so the variation comes only from the prompt. That said, we agree that off-axis change should be quantified, so we measure it for both generation and editing. We also have more discussions on how the strength control which the user sets, controls for this a bit, and some diversity is good because it helps with creative exploration (but more on this below).

Generation: We compute the TPIPS distance of the set conditioned on (i) the requested axis and (ii) other axes related to the prompt. When constructing our training and validation data, we ask an LLM to propose 3–5 plausible axes of variation per prompt; we use the first one for (i) and the rest for (ii). We test this across multiple diversity strengths α (the scale of the trained adapter; α = 0 is the untrained policy) and baselines. We report results for the fastest model in our setup (FLUX.2 [klein]) on validation prompts:

MethodTPIPS diversity gain (requested axis)TPIPS diversity gain (other axes)Pairs with more gain on requested axis
T-DIG α = 00.190.0196.9%
T-DIG α = 0.250.240.0597.4%
T-DIG α = 0.50.280.0996.4%
T-DIG α = 0.750.290.1197.4%
T-DIG α = 10.310.1395.9%
CADS (baseline)0.120.1248.5%
Group Inference (baseline)0.050.0446.9%

The diversity gain is larger on the requested axis than on the other axes for 96–97% of pairs at every strength. The baselines, which do not see the axis, spread their gain evenly across requested and non-requested axes, so they gain more on the requested axis for only 47–49% of pairs, i.e., chance.

Note that off-axis diversity is not expected to be zero. Axes of the same prompt can overlap in meaning: for example, if background is the requested axis and lighting is one of the other axes, changing one naturally changes the other. We also view some off-axis variation as a good thing: our goal is to show users a variety of outputs that help them explore different interpretations of a prompt, and off-axis creativity helps with that. Importantly, off-axis variation increases with diversity strength, which is a user control: at low strength there is almost no off-axis change (0.01 at α = 0), and at high strength there is some, so users can choose based on their preference.

Editing: In editing, the user starts from a set of images for a prompt and asks T-DIG to make them more diverse along one axis. T-DIG writes one edit instruction per image, and the editor applies it. We run the same test as above, but the diversity gain is measured relative to the source images (before editing) instead of an i.i.d. set.

Strength αTPIPS diversity gain (requested axis)TPIPS diversity gain (other axes)Pairs with more gain on requested axis
0 (untrained)0.200.0595.9%
0.250.230.0698.8%
0.50.240.0699.0%
0.750.250.0697.9%
10.250.0797.9%

At every strength, the diversity gain is 3.5–4.2× larger on the requested axis than on the other axes, and it is larger for 96–99% of pairs. Increasing the strength from 0 to 1 adds 0.05 on the requested axis and only 0.02 on the others, so off-axis change in editing is small and nearly flat across strengths.

We will add this analysis to the paper!

AuthorsOpenReview ↗

Weakness 2: Limited comparisons with existing prompt optimization methods

We'd like to note that the paper includes three such comparisons:

  1. PAG (Yun et al., 2025), a learned prompt-rewriting method for diverse generation, is a baseline in our main results.
  2. The untrained policy (T-DIG α = 0) is the same LLM with the same axis-conditioned instructions and no GRPO training. It is a zero-shot LLM prompt-expansion baseline, and it isolates what training adds.
  3. Prompt expansion with GPT-5.5 and Gemini 3.8 Flash (RQ8) tests whether a much larger general-purpose model, given the same instructions, closes the gap.

If the reviewer has other recommendations, we are happy to add them as well!

Thank you for the authors' response. I may have initially misunderstood some of the paper's claims and aspects of the experimental setup. After the clarification, my concerns have been addressed. Additionally, the authors' exploration of diversity is also commendable. I will keep my score unchanged.

Reviewer_SP5iOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+2)
Read review
Paper summary

This paper introduces T-DIG, a framework for text-conditioned diversity in image generation and editing. Unlike conventional diversity enhancement methods that maximize generic differences between generated images, T-DIG allows users to specify the desired axis of variation through natural language, such as artistic style, object attributes, or camera viewpoint.

Critical strengths
  1. The paper addresses a meaningful and practically relevant problem. Most existing diversity enhancement methods focus on increasing overall variation between generated images without considering which attributes users actually want to vary.
  2. The proposed approach is relatively simple and broadly applicable.
  3. The experiments cover multiple modern image generation models.
Critical weaknesses
  1. Important related work is missing. The paper emphasizes user-controlled image diversity as a central contribution and states that existing methods do not allow users to specify what should vary through free-form text. However, Varif.ai [1] has already investigated user-driven diversity control in image generation.
  2. The statistical significance claims in Table 1 appear questionable. Table 1 states that "All wins are significant (p < 0.05, sign test) except ‡ (p = 0.13)." However, the diversity-and-alignment win rates of T-DIG (α = 0.5) against T-DIG (α = 0) are 52.6%, 50.0%, and 47.6% for SD3.5 Large, SD3.5 Medium, and FLUX.2, respectively. Under a conventional one-sided sign test with a null hypothesis of a 50% win rate, results at or below 50% cannot provide statistically significant evidence that T-DIG (α = 0.5) is superior. Even the 52.6% win rate is unlikely to be statistically significant, given the limited number of comparisons for each model and baseline.
  3. The paper lacks an explicit evaluation of off-axis changes. The main objective is not only to increase diversity along the specified axis but also to preserve the original prompt and unrelated image attributes. However, the diversity reward Da(X)D_a(X) only measures variation along the specified axis, while Lalign(X,p)L_{\mathrm{align}}(X,p) primarily measures whether the generated images remain aligned with the original prompt. Neither metric directly ensures that attributes unrelated to the requested axis remain unchanged. For example, if the base prompt is "A cat sitting on a sofa" and the diversity axis is "The color of the cat's collar," changes to the cat's breed, the type of sofa, or the background may not be adequately reflected by these two metrics.
  4. The paper presents both generation and editing as important contributions, but it does not provide separate quantitative results specifically evaluating the editing policy. Moreover, the editing-specific drift penalty LdriftL_{\mathrm{drift}} is not adequately evaluated through ablation studies.

[1] Michelessa et al. Varif.ai: Vary and Verify User-Driven Diversity in Scalable Image Generation. ACM DIS, 2025.

Questions for authors
  1. Can the authors clarify the methodological novelty of T-DIG relative to Varif.ai?
5 comments
AuthorsOpenReview ↗

Weakness 1 / Question 1: Missing related work (Varif.ai) and methodological novelty

Thanks for pointing this out! We agree that Varif.ai is relevant to our work and should be cited. Like us, Varif.ai lets users decide which attributes should vary across a set of images and prompts a T2I model to generate different images. However, it differs from T-DIG in several ways:

1. Label lists vs. free-form text: In Varif.ai, each attribute (e.g., "color") must be expressed as a discrete list of labels (e.g., black, orange, blue; generated with an LLM), which are appended to the base prompt to form a new prompt for the T2I model. In T-DIG, the user can enter any free-form axis, which gives them more flexibility in what they want to change. Requests like "how he's feeling about summer" or "crazy endings that would surprise them" have no natural label list. Users can also ask for multiple things at once, and here Varif.ai's attribute generation breaks down -- for the axis "squirrel size, squirrel color, helicopter color, terrain" (Figure 3), the appended labels ("…, large, purple, red, desert") do not say which color belongs to which object.

2. Training-free prompting vs. a trained policy: Varif.ai prompts an off-the-shelf LLM without training, whereas we train the policy on a set-level diversity reward with alignment and preference penalties. Our α = 0 row is the closest prompting-only baseline, and training raises its gain over i.i.d. from +0.08–0.14 to +0.14–0.26 diversity at similar alignment (Table 3).

3. New dataset and trained models: We release a new dataset for axis-conditioned T2I diversity and trained checkpoints!

Direct comparison: We ran Varif.ai on the generation requests in our test set, using their code and prompt template (⟨prompt⟩, ⟨attribute⟩ ⟨label⟩), a uniform distribution over the proposed labels (in their paper users set this), and the same LLM as our policy (Qwen3.5 9B). Varif.ai uses a different initial noise per image. We report TPIPS diversity / VQAScore alignment:

ModelI.I.D.Varif.aiT-DIG α = 0T-DIG α = 0.5T-DIG α = 1
Qwen-Image0.12 / 0.890.19 / 0.890.27 / 0.860.37 / 0.860.38 / 0.86
SD3.5 Large0.20 / 0.830.25 / 0.820.28 / 0.800.36 / 0.840.35 / 0.83
SD3.5 Medium0.21 / 0.800.26 / 0.810.29 / 0.820.36 / 0.790.36 / 0.81
FLUX.2 [klein]0.14 / 0.910.25 / 0.880.28 / 0.880.37 / 0.870.39 / 0.87

Relative to i.i.d., Varif.ai raises diversity along the requested axis to 1.2–1.7× across models, the untrained T-DIG policy (α = 0) to 1.4–2.2×, and T-DIG (α = 1) to 1.7–3.1× (computed from unrounded scores). T-DIG (α = 1) is higher than Varif.ai by a further +0.10–0.19 and is more diverse on 90–97% of requests, at comparable alignment (VQAScore within 0.03). The untrained policy (α = 0) is also more diverse than Varif.ai on every model, which suggests that rewriting the prompt is more effective than appending a label, and training then adds most of the gain. We will add this discussion and the table to the paper!

AuthorsOpenReview ↗

Weakness 2: Statistical significance claims in Table 1

Apologies for the confusion. The numbers are correct, but they are computed while accounting for ties, which our caption did not explain. Each comparison has three possible outcomes: T-DIG wins, the baseline wins, or a tie (no majority of raters for either method). Ties are more frequent in the diversity + alignment setting, since a win there requires diversity to be greater and alignment to be greater than or equal to the baseline.

The percentage in Table 1 is T-DIG wins divided by all comparisons, ties included. A tie therefore lowers the reported win rate even though T-DIG did not lose that comparison; for example, 10 wins, 10 ties and 1 loss is reported as 10/21 = 47.6%. The sign test is two-sided and compares wins against losses only. A win rate at or below 50% therefore does not mean T-DIG lost more often than it won. For the three cells you mention (T-DIG α = 0.5 vs. α = 0, diversity + alignment):

Model (reported win rate)WinsTiesLossesp (two-sided sign test)
SD3.5 Large (52.6%)10720.039
SD3.5 Medium (50.0%)7700.016
FLUX.2 [klein] (47.6%)101010.012

We will replace the win percentages with wins / ties / losses for every cell (full table below) and state these details clearly in the text. Please let us know if we can clarify further.

CriterionModelvs. I.I.D.vs. CADSvs. Group Inf.vs. T-DIG α = 0
More variedQwen-Image23 / 2 / 0 (p<0.001)25 / 1 / 0 (p<0.001)22 / 1 / 2 (p<0.001)18 / 4 / 2 (p<0.001)
More variedSD3.5 Large21 / 2 / 0 (p<0.001)18 / 2 / 3 (p=0.001)18 / 4 / 1 (p<0.001)15 / 1 / 3 (p=0.008)
More variedSD3.5 Medium13 / 0 / 1 (p=0.002)12 / 2 / 0 (p<0.001)9 / 4 / 1 (p=0.021)9 / 5 / 0 (p=0.004)
More variedFLUX.2 [klein]20 / 1 / 0 (p<0.001)19 / 2 / 2 (p<0.001)20 / 2 / 1 (p<0.001)13 / 7 / 1 (p=0.002)
More varied + alignedQwen-Image17 / 8 / 0 (p<0.001)21 / 5 / 0 (p<0.001)19 / 6 / 0 (p<0.001)14 / 10 / 0 (p<0.001)
More varied + alignedSD3.5 Large19 / 4 / 0 (p<0.001)14 / 7 / 2 (p=0.004)13 / 9 / 1 (p=0.002)10 / 7 / 2 (p=0.039)
More varied + alignedSD3.5 Medium8 / 5 / 1 (p=0.039)9 / 5 / 0 (p=0.004)6 / 7 / 1 (p=0.125)7 / 7 / 0 (p=0.016)
More varied + alignedFLUX.2 [klein]13 / 8 / 0 (p<0.001)16 / 6 / 1 (p<0.001)16 / 6 / 1 (p<0.001)10 / 10 / 1 (p=0.012)
AuthorsOpenReview ↗

Weakness 3: Explicit evaluation of off-axis changes

We briefly discuss this in L200–204 as motivation for the length penalty, which we qualitatively observe helps control off-axis leakage. Further, all images are generated from the same noise latent, which ensures that the variations come only from the prompt. Nonetheless, we quantify this behavior with experiments for both generation and editing.

Generation: We compute the TPIPS distance of the set conditioned on (i) the requested axis and (ii) other axes related to the prompt. When constructing our training and validation data, we ask an LLM to propose 3–5 plausible axes of variation per prompt; we use the first one for (i) and the rest for (ii). We test this across multiple diversity strengths α (the scale of the trained adapter; α = 0 is the untrained policy) and baselines. We report results for the fastest model in our setup (FLUX.2 [klein]) on validation prompts:

MethodTPIPS diversity gain (requested axis)TPIPS diversity gain (other axes)Pairs with more gain on requested axis
T-DIG α = 00.190.0196.9%
T-DIG α = 0.250.240.0597.4%
T-DIG α = 0.50.280.0996.4%
T-DIG α = 0.750.290.1197.4%
T-DIG α = 10.310.1395.9%
CADS (baseline)0.120.1248.5%
Group Inference (baseline)0.050.0446.9%

The diversity gain is larger on the requested axis than on the other axes for 96–97% of pairs at every strength. The baselines, which do not see the axis, spread their gain evenly across requested and non-requested axes, so they gain more on the requested axis for only 47–49% of pairs, i.e., chance.

Note that off-axis diversity is not expected to be zero. Axes of the same prompt can overlap in meaning: for example, if background is the requested axis and lighting is one of the other axes, changing one naturally changes the other. We also view some off-axis variation as a good thing: our goal is to show users a variety of outputs that help them explore different interpretations of a prompt, and off-axis creativity helps with that. Importantly, off-axis variation increases with diversity strength, which is a user control: at low strength there is almost no off-axis change (0.01 at α = 0), and at high strength there is some, so users can choose based on their preference.

Editing: In editing, the user starts from a set of images for a prompt and asks T-DIG to make them more diverse along one axis. T-DIG writes one edit instruction per image, and the editor applies it. We run the same test as above, but the diversity gain is measured relative to the source images (before editing) instead of an i.i.d. set.

Strength αTPIPS diversity gain (requested axis)TPIPS diversity gain (other axes)Pairs with more gain on requested axis
0 (untrained)0.200.0595.9%
0.250.230.0698.8%
0.50.240.0699.0%
0.750.250.0697.9%
10.250.0797.9%

At every strength, the diversity gain is 3.5–4.2× larger on the requested axis than on the other axes, and it is larger for 96–99% of pairs. Increasing the strength from 0 to 1 adds 0.05 on the requested axis and only 0.02 on the others, so off-axis change in editing is small and nearly flat across strengths.

We will add this analysis to the paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.