acceptodds
Under review as a conference paper at ICLR 2027

Mine, Sample, Render: Test-Time Diversity Alignment of Generative Models

Abstract

Modern generative models produce high-quality individual outputs but struggle to sample diversely, even when explicitly instructed to do so. We introduce Mine-Sample-Render (MSR), a test-time approach that externalizes the diversity into an explicit, corpus-derived sampling procedure. Our pipeline (i) mines an interpretable categorical schema from a corpus of real artifacts and supplies statistics for sampling; (ii) samples attribute configurations from the corpus-derived statistics; (iii) uses a generative model purely as a conditional renderer of a sampled configuration. Across weather report generation, user simulation, webpage design, text-to-image generation, and agent-generated 3D assets, we evaluate the diversity and output quality of MSR-guided generation and show that MSR improves alignment with corpus distributions and diversity of generated outputs.

Primary area:
generative models
Keywords:
generative models, sampling, diversity

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_i2ryOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mine–Sample–Render (MSR), a training-free framework for improving generative diversity. MSR mines interpretable attributes from a reference corpus, estimates their statistical dependencies, and samples attribute configurations to guide generation. Experiments across five domains demonstrate improvements in output diversity and distributional alignment.

Critical strengths
  1. MSR improves diversity across text, images, webpages, user simulation, and 3D asset generation without retraining the generators.

  2. Extensive experiments demonstrate consistent gains over prompt-only and prompt-rewriting baselines, particularly in image generation and user simulation.

Critical weaknesses
  1. The framework primarily combines LLM-based attribute extraction, conventional statistical sampling, and conditional generation. Its technical distinction from existing attribute-guided and structured generation methods is limited.

  2. Most experiments measure diversity rather than fidelity to the reference distribution. Higher Vendi Scores do not necessarily indicate better distributional alignment. Moreover, several evaluations rely on LLM judges, raising concerns about evaluation bias.

Questions for authors
  1. Can the authors compare MSR with simpler alternatives, such as independently sampling mined attributes or directly resampling observed attribute configurations, to isolate the benefit of modeling attribute dependencies?

  2. Can the authors evaluate distributional alignment on real-world datasets beyond the weather and user-simulation tasks, rather than relying primarily on diversity metrics?

4 comments
AuthorsOpenReview ↗

Response to Reviewer i2ry (1/2)

Thank you for the constructive review, and for recognizing that MSR improves diversity across five domains without retraining, with consistent gains over prompt-only and rewriting baselines. Below we address each of your concerns.

1. Technical distinction from attribute-guided and structured generation (W1)

We agree that MSR's components are individually known. We see the contribution in the separation itself: identifying that the generator is a poor sampler, and showing that an explicit, interpretable, corpus-derived sampler fixes this across very different domains and for black-box models and agents. This differs from existing attribute-guided and structured generation methods in three ways:

  • Attributes and statistics come from data, not from the LLM's prior. Attribute-guided methods such as AttrPrompt let an LLM propose attributes and sample them independently. MSR mines attributes from a reference corpus and samples from their estimated joint distribution. In a new comparison in user simulation, AttrPrompt, whose attributes are proposed by an LLM, does not align well with real users (Table in Section 3 below).
  • It works with any black-box generator or agent. Because MSR only changes the generator's input, it requires no access to model internals or decoding logits and applies equally to proprietary image models, LLMs, and tool-using agents, which is what makes a single method usable across all five domains.
  • Separation can mitigate the diversity-quality trade-off. Making the generator itself more diverse costs quality. For the 3D chairs (Table 4), coordinated prompting lowers quality from 82.35 to 70.80, while MSR is more diverse and keeps quality at 83.55; for text-to-image, our results suggest that the diversity MSR adds does not come at a high cost in realism.

2. Fidelity to the reference distribution and LLM judges (W2 / Q2)

We agree that reference-free diversity metrics alone cannot support the alignment claim: a uniform sampler over the schema can increase Vendi while moving outputs away from the corpus. This is why our default samplers target the corpus distribution, and why alignment has to be measured directly. Beyond weather and user simulation, we now report reference-based alignment metrics in text-to-image and 3D chairs:

Text-to-image (GPT Image 2). We report reference-based metrics against held-out COCO images (500 images per method; validity on 100 images per method):

MethodVendi (CLIP) ↑Vendi (DINOv2) ↑FD-DINOv2 ↓KID ↓CMMD ↓Validity ↑
Prompt only2.372.160.9370.01630.860100.0%
Independent rewrite3.023.190.8630.01640.91599.0%
Coordinated rewrite3.864.320.7860.01120.83799.0%
MSR-Direct5.144.970.7190.00570.85499.0%
MSR-Resolved6.686.730.6390.00841.06097.0%
MSR-Direct + uniform sampler6.055.290.6970.01020.99296.0%

On the DINOv2- and Inception-based metrics, MSR's added diversity moves outputs closer to real images: MSR-Resolved gives the best FD-DINOv2, and MSR-Direct the best KID. KID is unbiased at small sample sizes, which makes it the more reliable of the two here. CMMD gives a more mixed picture: MSR-Direct is on par with the baselines, while MSR-Resolved scores worst. The uniform sampler, compared with MSR-Direct, is more diverse but has worse KID, worse CMMD, and the lowest validity of all methods.

3D chairs. We also computed reference-based metrics for the chairs, against the corpus of reference designs. We used two weightings (equal mass per design family and equal mass per design), and both image embeddings (DINOv2) and text embeddings of chair descriptions. Each chair was described by GPT-5.6 Sol, and the description was embedded with text-embedding-3-large:

Metric (equal family mass)MSRIndependentCoordinated
FD (image) ↓0.7600.7931.168
FD (text) ↓0.1680.2240.201
KID ↓0.0580.0720.063
CMMD ↓2.443.183.05
MMD, DINOv2 (RBF) ↓0.1450.1680.253
MMD, text (RBF) ↓0.0580.1650.099

MSR is closest to the reference distribution on every metric, and the same holds with equal per-design weighting (e.g., KID 0.058 vs. 0.069 and 0.066; CMMD 2.40 vs. 3.19 and 3.07). Notably, coordinated generation, which asks the agent itself to diversify, moves further from the reference than independent generation on image FD and MMD. MSR's diversity, by contrast, stays aligned with the corpus.

AuthorsOpenReview ↗

Response to Reviewer i2ry (2/2)

Addressing LLM-judge bias. Beyond LLM judges, every central claim is also supported by metrics that involve no judge: diversity is measured with embedding-based Vendi scores, and alignment with real data is measured with reference-based metrics (KL/JS in weather and user simulation, and FD, KID, CMMD, and MMD in images and 3D chairs). In addition, we reran the image odd-one-out evaluation as balanced 10-vs-10 pairwise comparisons (10 MSR-Resolved images vs. 10 images from one baseline per prompt; removed images receive ranks 1–10 and survivors rank 20) using both GPT-5.6 Sol and an independent judge, Kimi K3, which was not involved in generation, rewriting, or resolution:

JudgeComparatorMSR mean rank ↓Comparator mean rankPaired gap [95% CI]Prompts with a lower MSR mean removal score
GPT-5.6 SolPrompt only5.5020.0014.50 [14.50, 14.50]50/50
GPT-5.6 SolIndependent rewrite5.9219.5813.67 [13.14, 14.10]50/50
GPT-5.6 SolCoordinated rewrite8.6916.818.12 [6.84, 9.30]45/50
Kimi K3Prompt only5.5519.9514.40 [14.24, 14.50]50/50
Kimi K3Independent rewrite5.9119.5913.67 [13.12, 14.13]50/50
Kimi K3Coordinated rewrite10.0515.455.40 [3.84, 6.88]45/50

With equal pools, MSR is judged more diverse than every baseline by both judges, with confidence intervals well away from zero (50/50 prompts against prompt-only and independent rewriting, and 45/50 against coordinated rewriting for both judges). The independent judge leads to the same conclusions as the original one.

3. Isolating the benefit of modeling attribute dependencies (Q1)

We thank the reviewer for this suggestion. The two samplers in the paper, ancestral sampling and topic-model sampling, are a starting point we chose, as the most natural default target is the corpus distribution. Any sampler over the schema is an instance of MSR: ancestral sampling, independent empirical-marginal sampling, directly resampling observed configurations, and uniform sampling, with the mining and rendering stages unchanged. This flexibility comes directly from MSR separating sampling from rendering: in prompt-only, prompt-rewriting, or persona baselines, the generator also acts as the sampler, so its output distribution cannot be inspected, edited, or swapped for a different statistical sampler. We make this framing more explicit in the introduction.

Following your suggestion, we compare these samplers with MSR alongside the non-MSR baselines in user simulation (GPT-5.5) on the common intersection of N=4,721N = 4,721 valid conversations across all 10 variants (with PCA and k-means, k=500k = 500, refit on each real–simulated pair). Masked refers to using the compatibility mask for sampling:

Method (GPT-5.5, N=4,721N = 4,721)Forward KL ↓JS ↓Coverage ↑
Prompt only0.5280.5340.595
PersonaHub0.5260.5390.567
AttrPrompt (GPT-5-mini schema, uniform sampling)0.5610.5770.438
MSR fine-grained, uniform sampler0.4550.4620.609
MSR compact, ancestral sampler0.3210.3200.762
MSR fine-grained, ancestral sampler0.3140.3120.766
MSR fine-grained, independent marginals (unmasked)0.3070.3010.776
MSR fine-grained, independent marginals (masked)0.3050.3080.748
MSR fine-grained, observed-row resampling (unmasked)0.2790.2830.773
MSR fine-grained, observed-row resampling (masked)0.2940.2980.774

Every MSR sampler over the mined schema, even simple uniform sampling (0.455 / 0.462), improves over Prompt only, PersonaHub, and AttrPrompt (0.561 / 0.577), showing the value of mining attributes from real data rather than eliciting them from the LLM's prior. Within MSR, replacing uniform sampling with any sampler that respects empirical corpus frequencies (ancestral, independent marginals, or observed-row resampling) reduces KL and JS by 0.14–0.18 and raises coverage from 60.9% to 74.8%–77.6%; uniform sampling even reduces coverage because many uniformly drawn combinations rarely occur in real users. Among the corpus-based samplers, independent empirical marginals (0.305–0.307) perform on par with ancestral sampling (0.314), though unmasked marginals produce incompatible attribute combinations on 33.7% of draws (vs. 0% when masked). Directly resampling observed rows achieves the lowest KL and JS (0.279 / 0.283 unmasked; 0.294 / 0.298 masked) because it preserves the exact empirical joint distribution, confirming that cross-attribute dependencies improve distributional fit at the cost of only replaying the training configurations rather than generalizing to unseen combinations as ancestral sampling does.

Thank you for the detailed response. The new distributional alignment evaluations and sampler ablations address most of my empirical concerns. However, independent marginal sampling performs comparably to ancestral sampling, suggesting that the benefit of explicitly modeling attribute dependencies remains limited. While the additional results strengthen the empirical contribution, my concern about technical novelty remains.

Thank you for your response.

In complex schemas, you cannot simply combine attributes independently from their marginals (we denote that sampling as IndepMarg): independent draws frequently pair mutually incompatible attributes (e.g., camera_close_up + view_long_shot; 49.4%–72.4% of independent image draws contain hard contradictions vs. 0.0% for Ancestral). Passing independent combinations to a generator causes one of two failures:

1. The generator obeys the incompatible combination →\rightarrow invalid samples and worse joint alignment: In weather report generation, independent marginal sampling degrades joint Forward KL, JS, and coverage, and causes the LLM to render 2.63-3.33% physically impossible reports (snow during warm/hot temperatures vs. 0.00% for Ancestral):

MethodFwd KL ↓JS ↓Coverage ↑Invalid configurations ↓
MSR + Independent marginals (global)0.4090.34470.4%3.33%
MSR + Independent marginals (per-city)0.1930.20866.5%2.63%
MSR + Ancestral (joint)0.1810.20368.5%0.00%

2. The generator rejects/drops incompatible attributes →\rightarrow worse diversity and worse distributional alignment: In text-to-image with GPT Image 2, the image renderer refuses to draw contradictory scenes and silently drops over 62% of independently sampled attributes (37.4% realized vs. 61.6% MSR-Direct and 80.2% MSR-Resolved). Independent marginal sampling reduces diversity measured with DINOv2 and worsens distributional alignment across all metrics (FD-DINOv2, KID, CMMD):

MethodControls Realized ↑Vendi (CLIP) ↑Vendi (DINOv2) ↑FD-DINOv2 ↓KID ↓CMMD ↓
MSR-Direct + independent marginals37.4%5.254.550.7260.00700.990
MSR-Direct (joint ancestral)61.6%5.144.970.7190.00570.854

(In User Simulation, IndepMarg and Ancestral only looked close on conversation KL [0.307 vs. 0.314] because following Mehri et al. [2026]'s evaluation embeds all concatenated facets into one vector, which acts as an additive 1D-marginal summary. On pairwise attribute co-occurrence, IndepMarg has -0.02 correlation with real users [33.7% incompatible draws] vs. 0.69–0.84 for Ancestral.)

On technical novelty: Our contribution is a unified Mine–Sample–Render framework, where each stage solves a distinct failure mode of prior synthetic generation:

  1. Stage 1 mining (our data-driven schema vs. LLM-prior schema): Establishing the axes of variation directly from data improves distributional alignment even under the exact same uniform sampling: compared to AttrPrompt (which also samples uniformly over LLM-proposed attributes), MSR + Uniform reduces Forward KL from 0.561 -> 0.455 in User Simulation (raising coverage from 43.8% -> 60.9%).
  2. Stage 2 sampling with empirical corpus statistics (empirical frequencies vs. uniform): Sampling from corpus statistics rather than uniformly (Uniform -> IndepMarg / Ancestral) further increases distributional alignment. In User Simulation it reduces Forward KL by 31%–33% (0.455 -> 0.307 / 0.314, raising coverage to 76.6%–77.6%), in Weather report generation by 68%–70% (0.603 -> 0.193 / 0.181, cutting invalid reports from 17.07% to 2.63% / 0.00%), whereas in text-to-image generation it improves image KID and CMMD (0.0102 -> 0.0070 / 0.0057 and 0.992 -> 0.990 / 0.854).
  3. Stage 2 joint dependency modeling: By Proposition 1 (see our response to Reviewer nTFD), end-to-end distributional error is bounded by sampler mismatch TV(q,p)\mathrm{TV}(q,p) plus renderer non-realization Pr⁡[a(x)≠c]\Pr[a(x)\ne c]. Sampling from the joint distribution rather than independent marginals preserves cross-attribute correlations (pairwise attribute co-occurrence r = 0.69–0.84 vs. -0.02 in User Simulation; Observed Rows KL 0.279 vs. 0.307) and prevents both sampler mismatch TV(q,p)\mathrm{TV}(q,p) (in weather report generation 0.00% invalid configurations vs. 2.63%–3.33%, reducing Joint KL from 0.193 -> 0.181) and renderer non-realization Pr⁡[a(x)≠c]\Pr[a(x)\ne c] (in text-to-image 61.6% realized controls vs. 37.4%, improving DINOv2 Vendi from 4.55 -> 4.97, and distributional alignment across all metrics: FD-DINOv2 from 0.726 -> 0.719, KID from 0.0070 -> 0.0057, and CMMD from 0.990 -> 0.854).
Reviewer_8M1pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Mine–Sample–Render (MSR), a training-free framework that improves generative diversity by mining attributes from reference corpora, sampling attribute configurations, and guiding generation. Experiments across five tasks demonstrate improvements in diversity and distributional alignment.

Critical strengths
  1. The paper addresses an important problem in generative diversity and presents a clear and intuitive framework.
  2. The experiments cover five diverse tasks and multiple generative models, showing promising improvements in diversity or distributional alignment.
Critical weaknesses
  1. Limited novelty. While the problem of diversity in generative models is important, the proposed approach is relatively straightforward and offers limited new insights. The paper largely demonstrates an intuitive idea—explicitly sampling diverse attributes leads to more diverse outputs—without revealing deeper or unexpected findings about generative diversity.
  2. Scalability and computational cost. MSR relies on a representative reference corpus and extensive LLM-based preprocessing.
  3. Insufficient ablation studies. The paper lacks sufficient ablations to isolate the contributions of individual components, particularly whether the proposed statistical sampling strategies outperform simpler alternatives such as independent or uniform attribute sampling.
  4. Limited analysis of corpus dependence. The effectiveness of MSR depend heavily on the quality and representativeness of the reference corpus, but the paper does not systematically investigate this sensitivity.
Questions for authors
  1. How does MSR compare with simpler attribute sampling strategies, such as uniform sampling?
  2. How does MSR compare with standard decoding strategies, such as temperature and top-p tuning, on models that support these controls?
  3. Could the authors provide comparisons with closely related methods, such as AttrPrompt and fLSA, on applicable tasks?
  4. Do MSR's diversity gains persist on more complex compositional text-to-image benchmarks without compromising prompt adherence?
  5. Could the authors quantify the preprocessing cost of MSR and analyze how it scales with corpus size and the number of attributes?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8M1p (1/4)

Thank you for the constructive review, and for finding the framework clear and intuitive and the experiments broad. Below we address each of your concerns.

1. Novelty and insights (W1)

We agree that MSR's components are individually known. We see the contribution in the separation itself: identifying that the generator is a poor sampler, and showing that an explicit, interpretable, corpus-derived sampler fixes this across very different domains and for black-box models and agents. This differs from existing attribute-guided and structured generation methods in three ways:

  • Attributes and statistics come from data, not from the LLM's prior. Attribute-guided methods such as AttrPrompt let an LLM propose attributes and sample them independently. MSR mines attributes from a reference corpus and samples from their estimated joint distribution. Similarly, generic persona banks such as PersonaHub do not reliably align with a target population. In a new comparison in user simulation, AttrPrompt, whose attributes are proposed by an LLM, does not align well with real users (Table in Section 3 below).
  • It works with any black-box generator or agent. Because MSR only changes the generator's input, it requires no access to model internals or decoding logits and applies equally to proprietary image models, LLMs, and tool-using agents, which is what makes a single method usable across all five domains.
  • Separation can mitigate the diversity-quality trade-off. Making the generator itself more diverse costs quality. For the 3D chairs (Table 4), coordinated prompting lowers quality from 82.35 to 70.80, while MSR is more diverse and keeps quality at 83.55; for text-to-image, our results suggest that the diversity MSR adds does not come at a high cost in realism.

2. Scalability and computational cost (W2 / Q5)

We logged the LLM requests and reconstructed the token usage needed to build the user-simulation schemas from the saved prompts and outputs:

SchemaLLM requests (GPT-5-mini)Input tokensOutput tokens (saved JSON)
Compact (84 attributes, 600 conversations)2,1837.46M1.62M
Fine-grained (465 attributes, same 600 conversations)2,31317.86M1.24M

Building a schema is a modest, one-time investment per reference corpus (~2,200–2,300 GPT-5-mini calls and <20M input/output tokens, a relatively small one-time token cost for an entire corpus, especially compared with generating thousands of downstream outputs). Once built, the schema and corpus statistics can be reused across many generations and different generators without repeating the mining and labeling stages, amortizing the upfront cost. The sampling cost afterward is negligible compared to the cost of generating the outputs.

Scaling with corpus size NN and attribute count KK. Schema mining processes documents in fixed-size batches with periodic consolidation, so mining calls scale as O(N)O(N) and are largely insensitive to the final number of attributes (483 calls for the 84-attribute compact schema vs. 513 for the 465-attribute fine-grained schema). Backfill labeling evaluates each corpus item against the schema in a fixed number of passes per document (here, 3 passes × 600 conversations = 1,800 calls for both schemas), so the number of labeling calls scales linearly in NN and stays constant as KK grows by 5.5× (84 → 465), while input tokens per call grow linearly with the schema description length.

AuthorsOpenReview ↗

Response to Reviewer 8M1p (2/4)

3. Ablations and comparisons with related methods (W3 / Q1 / Q3): AttrPrompt, fLSA, uniform sampling

We thank the reviewer for this suggestion. The two samplers in the paper, ancestral sampling and topic-model sampling, are a starting point we chose, as the most natural default target is the corpus distribution. Any sampler over the schema is an instance of MSR: ancestral sampling, independent empirical-marginal sampling, directly resampling observed configurations, and uniform sampling, with the mining and rendering stages unchanged. This flexibility comes directly from MSR separating sampling from rendering: in prompt-only, prompt-rewriting, or persona baselines, the generator also acts as the sampler, so its output distribution cannot be inspected, edited, or swapped for a different statistical sampler. We make this framing more explicit in the introduction.

To isolate the roles of the mined schema and the sampler, we compare AttrPrompt (attributes proposed by GPT-5-mini from its own prior and sampled uniformly, without a corpus) and uniform sampling with the two existing MSR samplers over the mined schema in user simulation (GPT-5.5, N=4,721N = 4,721 common valid conversations across all arms; PCA and k-means, k=500k = 500, refit on each real–simulated pair):

Method (GPT-5.5)Forward KL ↓JS ↓Coverage ↑
Prompt only0.5280.5340.595
PersonaHub0.5260.5390.567
AttrPrompt (GPT-5-mini schema, uniform sampling)0.5610.5770.438
MSR fine-grained, uniform sampler0.4550.4620.609
MSR fine-grained, independent empirical marginals0.3050.3080.748
MSR compact, ancestral sampler0.3210.3200.762
MSR fine-grained, ancestral sampler0.3140.3120.766

We also tested uniform sampling over the mined schema in text-to-image generation (GPT Image 2; 500 images per method; validity on 100 images per method):

Method (GPT Image 2)Vendi (CLIP) ↑Vendi (DINOv2) ↑Validity ↑
Prompt only2.372.16100.0%
Independent rewrite3.023.1999.0%
Coordinated rewrite3.864.3299.0%
MSR-Direct (topic-model sampler)5.144.9799.0%
MSR-Resolved (topic-model sampler)6.686.7397.0%
MSR-Direct + uniform sampler6.055.2996.0%

Together, these results separate the contributions of the mined schema and the corpus-aligned sampler:

  • Mined schema vs. LLM prior (AttrPrompt): Uniform sampling over the mined schema (0.455 / 0.462 / 0.609) improves substantially on prompt-only generation, PersonaHub, and AttrPrompt (0.561 / 0.577 / 0.438), showing the value of mining attributes from real data rather than eliciting them from the LLM's prior.
  • Corpus frequencies and dependencies vs. uniform sampling: Accounting for empirical marginal frequencies (0.305 / 0.308 / 0.748) or full ancestral dependencies (0.314 / 0.312 / 0.766) reduces forward KL and JS by ~0.15 over uniform sampling (0.455 / 0.462) and raises coverage from 60.9% to 74.8%–76.6%. Without hierarchical applicability masking, independent marginal sampling violates structural parent-child applicability rules on 33.7% of draws because it ignores cross-attribute dependencies, whereas ancestral sampling respects them by construction and achieves higher coverage (76.6% vs. 74.8%). In text-to-image, uniform sampling increases raw Vendi over MSR-Direct (5.14 → 6.05 CLIP; 4.97 → 5.29 DINOv2) by pairing rare attribute combinations, at a modest cost in validity (96.0% vs. 99.0%).
  • Relationship to fLSA: As discussed in Sec. 1 (p. 2) and App. A (p. 16), fLSA (Xu et al., 2025) is a close precursor that also induces natural-language latent variables from a corpus and samples them before rendering, but it models a sequential text outline: each document segment is assigned a single 1-of-KK cluster tag via EM-style prompting, and a bigram transition model is fit over tag sequences. We instead choose to target domains (webpages, images, 3D chairs, and user profiles), where an artifact is characterized not by a sequence of 1-of-KK segment tags, but by a compositional configuration of dozens to thousands of simultaneously active attributes (84–5,484 attributes per domain) with cross-attribute dependencies and compatibility constraints.
AuthorsOpenReview ↗

Response to Reviewer 8M1p (3/4)

4. Standard decoding strategies: temperature and top-p (Q2)

We thank the reviewer for this suggestion. We ran a seven-condition decoding comparison for Kimi K2.6 on webpage generation (10 pages per prompt across the 10 Website100 prompts with the lowest baseline diversity = 100 draws per condition), comparing default prompt-only decoding (temperature and top_p omitted), two temperature settings (T∈{0.6,1.3}T \in \{0.6, 1.3\}), two nucleus-sampling settings (top-p∈{0.8,1.0}\text{top-}p \in \{0.8, 1.0\}), and MSR with both webpage schemas:

Kimi K2.6, webpages (10-prompt subset)Vendi (text) ↑Vendi (image) ↑
Prompt only, default decoding2.792.78
Prompt only, T=0.6T = 0.62.692.64
Prompt only, T=1.3T = 1.3Invalid generationsInvalid generations
Prompt only, top-p=0.8\text{top-}p = 0.82.702.67
Prompt only, top-p=1.0\text{top-}p = 1.02.773.00
MSR + WebSight4.505.68
MSR + Design2Code3.803.98

Varying temperature and top-pp within the valid regime (T=0.6T = 0.6, top-p∈{0.8,1.0}\text{top-}p \in \{0.8, 1.0\}) changes both structural and visual diversity only modestly around the default baseline (2.69–2.79 text Vendi and 2.64–3.00 image Vendi). By contrast, MSR raises structural Vendi to 4.50 and visual Vendi to 5.68 with WebSight, and to 3.80 and 3.98 with Design2Code, with positive visual Vendi gains on all 10/10 matched tasks. Meanwhile, raising the temperature to T=1.3T = 1.3 causes Kimi K2.6 to generate broken or empty outputs rather than valid webpages: only 1 of 100 generations passed HTML validation, while the remaining 99 terminated with either empty or incomplete/malformed HTML lacking a complete document structure, even after retries. Decoding controls act at the token level during rendering, whereas MSR varies high-level design decisions explicitly before rendering and leaves the generator's decoding strategy at a reliable operating point. Finally, many black-box generators and APIs, including proprietary image models, do not expose temperature or top-pp controls at all.

5. Prompt adherence and compositional text-to-image generation (Q4)

To assess whether conditioning on sampled attributes compromises prompt adherence or visual quality in text-to-image generation, we extended the paper's validity evaluation (App. B.5.7, Table 14), which covers only MSR, by judging the baselines with the same judge and protocol (validity = the image satisfies the request and is visually acceptable; % of images):

GeneratorPrompt onlyIndependent rewriteCoordinated rewriteMSR-DirectMSR-Resolved
GPT Image 210099999997
MAI-Image-2.510095989697
FLUX.2-flex9998999590

For GPT Image 2 and MAI-Image-2.5, MSR's validity is within 1–4 points of the baselines (1–4 images out of 100), and on MAI-Image-2.5 both MSR variants are above independent rewriting. The exception is MSR-Resolved with FLUX.2-flex, where validity drops to 90%. Here, resolution trades some validity for attribute realization (95% → 90% validity, 71.1% → 77.9% attributes realized), so for this generator MSR-Direct (95%) is the better operating point when validity matters most. We hypothesize that FLUX.2-flex's text encoder struggles with the long, attribute-rich prompts that resolution produces.

For compositional prompts, MSR is specifically designed to diversify underspecified degrees of freedom without overriding explicit prompt constraints:

  1. Strict prompt priority and compatibility filtering: Under MSR's rendering contract (Sec. 2.4), the user's request always takes priority over sampled attributes; sampled attributes act only as non-exhaustive guidance on open degrees of freedom, and any sampled attribute that conflicts with the prompt is dropped.
  2. Two-stage semantic resolution: In MSR-Resolved, an LLM resolver explicitly binds role placeholders (X,YX, Y) to compatible entities and reconciles sampled attributes with the prompt into a single coherent scene plan before calling the image generator. Across 2,000 audited attribute occurrences per generator (Table 14), semantic resolution reconciles 99.2% of sampled controls and raises realized attributes from 61.6% to 80.2% on GPT Image 2 (65.3% → 78.5% on MAI-Image-2.5; 71.1% → 77.9% on FLUX.2-flex) while maintaining 97% prompt-satisfaction and visual validity on GPT Image 2 and MAI-Image-2.5.
Reviewer_ueCBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Mine–Sample–Render (MSR), a training-free approach for test-time diversity alignment. An LLM mines binary attributes by comparing small cohorts of a reference corpus, attribute configurations are sampled from their estimated joint distribution, and the generator renders the output conditioned on the request and the sampled attributes. The approach is simple and interpretable, and the controlled weather-report and user-simulation studies are convincing.

Critical strengths
  • The idea of externalizing diversity into an explicit, corpus-derived sampling step is simple, training-free, and applicable to black-box generators, including tool-using agents.
  • The mined schema is interpretable, which makes it clear what varies in the generated outputs and how often.
  • The controlled weather-report study shows that the mining stage recovers most of the true axes of variation (13 of 16 attributes), and that MSR substantially reduces the distributional gap.
  • The user-simulation experiments follow an established protocol (Mehri et al., 2026) and show consistent improvements across five simulators.
  • The experiments cover a wide range of domains, and the implementation details are thoroughly documented.
Critical weaknesses

I have some concerns about whether the evaluation supports the central claim of diversity alignment. My main concerns are as follows.

1. The evaluation measures diversity rather than alignment with the reference distribution.

The goal of the paper is to bring the generated distribution closer to the real-data distribution, but only the weather and user-simulation experiments compare against a reference distribution. For webpages, images, and 3D assets, the evaluation relies only on reference-free measures (Vendi score and odd-one-out). To see why this is not enough, consider a sampler that draws attributes uniformly at random from the schema: it would likely obtain a higher Vendi score while moving the outputs away from the corpus. The ablation in Appendix B.5.8 points in this direction, as a higher topic weight, which preserves fewer corpus dependencies, monotonically increases the Vendi score. I believe reference-based metrics such as precision/recall [4], density/coverage [5], or FD-DINOv2 [6] are essential to support the alignment claim.

2. The diversity evaluation itself is not reliable enough.

Each Vendi score uses only ten samples with an RBF kernel, so it is bounded by 10 and depends strongly on the bandwidth, which is not reported and, according to Appendix B.5.4, was selected on the evaluated data. The authors should report it, show that the rankings hold across bandwidths, and add confidence intervals. The odd-one-out evaluation is also confounded: in the image experiment, 30 of the 40 pooled images come from three similar baselines, so any distinct-looking group of ten would be removed first, even if its images were similar to each other. Also, the judge is the model used for MSR resolution and rewriting (GPT-5.6 Sol), or the generating agent itself (GPT-6 Astra).

3. Missing test-time diversity baselines.

For text-to-image generation, the only baselines are prompt rewriting. Training-free methods such as CADS [2], Particle Guidance [3], and SPARKE [1], which provides prompt-aware diversity guidance, already increase diversity at inference time. Although several of the evaluated generators are proprietary, a comparison on an open model (e.g., FLUX.1-dev) would clarify whether MSR offers an advantage beyond diversity guidance, or whether the two approaches are complementary. These works should be discussed and compared.

4. Quality evaluation and internal inconsistencies.

Validity and quality are assessed only by LLM judges. For text-to-image generation, validity is not reported for the baselines, so the quality cost of MSR cannot be assessed, and for 3D assets a single agent judge scores 20 chairs per method. Moreover, the quality scores in Table 4 and Table 15 differ (e.g., 83.55 vs. 84.70 for MSR), and Appendix B.6.4 and the caption of Table 15 disagree on who assigned the criterion weights.

5. Novelty and reproducibility.

As the authors acknowledge, the individual components are known, so the contribution lies mainly in their combination, which makes the evaluation especially important. The pipeline also relies on proprietary models, the cost of mining and labeling is not reported, and labeling accuracy is verified only in the synthetic weather setting.


[1] Jalali et al. SPARKE: Scalable Prompt-Aware Diversity Guidance in Diffusion Models via RKE Score. NeurIPS 2025.

[2] Sadat et al. CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling. ICLR 2024.

[3] Corso et al. Particle Guidance: Non-I.I.D. Diverse Sampling with Diffusion Models. ICLR 2024.

[4] Kynkäänniemi et al. Improved Precision and Recall Metric for Assessing Generative Models. NeurIPS 2019.

[5] Naeem et al. Reliable Fidelity and Diversity Metrics for Generative Models. ICML 2020.

[6] Stein et al. Exposing Flaws of Generative Model Evaluation Metrics and Their Unfair Treatment of Diffusion Models. NeurIPS 2023.

Questions for authors
  • Does MSR bring the generated images and webpages closer to the reference corpus, or does it only increase diversity? Can you report precision/recall, density/coverage, or FD-DINOv2 against the corpus?
  • Which RBF bandwidths are used for the Vendi scores, how are they chosen, and do the rankings hold across bandwidths?
  • How does MSR compare with test-time diversity guidance methods such as CADS, Particle Guidance, and SPARKE on an open text-to-image model, and can they be combined?
  • Can the odd-one-out evaluation be repeated with pairwise comparisons of equal pool sizes and a judge that is not involved in generation?
  • What are the validity rates of the baselines in the text-to-image experiment, and how do you explain the differences between Table 4 and Table 15?
  • How accurate is the corpus labeling outside the weather setting, and what is the cost of mining and labeling?
4 comments
AuthorsOpenReview ↗

Response to Reviewer ueCB (1/4)

Thank you for this detailed and constructive review, and for recognizing that MSR is simple, training-free, interpretable, and applicable to black-box generators and agents. We address all your concerns below.

1. Alignment with the reference distribution

We agree that reference-free diversity metrics alone cannot support the alignment claim, and we thank the reviewer for the uniform-sampler example, which captures the issue precisely. The core idea of MSR is to separate sampling from rendering: generators are excellent renderers but poor samplers, so diversity comes from an explicit sampler over the mined schema. A uniform sampler is one valid choice within MSR, and, as the reviewer points out, it can increase diversity while moving outputs away from the corpus. This is why our default samplers target the corpus distribution, and why alignment has to be measured directly. We now do so in three settings.

User simulation (uniform vs. corpus-based sampling, GPT-5.5).

MethodForward KL ↓JS ↓Coverage ↑
Prompt only0.5180.5360.567
PersonaHub0.5220.5370.562
MSR compact, ancestral sampler0.3180.3190.747
MSR fine-grained, ancestral sampler0.3060.3080.773
MSR fine-grained, uniform sampler0.4590.4660.616

To compare all methods on exactly the same conversations, we re-evaluated every method on the 4,796 conversations available for all simulated variants (vs. 4,903 in the paper); the rest were dropped because the simulator refused some sampled attribute combinations. Metrics are computed separately for each real–simulated pair, with PCA and k-means (k = 500) refit on each pair, so adding new methods does not change how the existing ones are evaluated. The values for the methods already in the paper therefore differ slightly from Table 1, only because the intersection is 107 conversations smaller and the clustering is refit on it.

With the same fine-grained schema, switching from ancestral to uniform sampling raises forward KL from 0.306 to 0.459 and JS from 0.308 to 0.466, and lowers coverage from 77.3% to 61.6%. Uniform sampling even reduces coverage, because many of its configurations are combinations real users rarely exhibit.

Text-to-image (GPT Image 2). We report reference-based metrics against held-out COCO images.

MethodVendi (CLIP) ↑Vendi (DINOv2) ↑FD-DINOv2 ↓KID ↓CMMD ↓Validity ↑
Prompt only2.372.160.9370.01630.860100.0%
Independent rewrite3.023.190.8630.01640.91599.0%
Coordinated rewrite3.864.320.7860.01120.83799.0%
MSR-Direct5.144.970.7190.00570.85499.0%
MSR-Resolved6.686.730.6390.00841.06097.0%
MSR-Direct + uniform sampler6.055.290.6970.01020.99296.0%

On the DINOv2- and Inception-based metrics, MSR's added diversity moves outputs closer to real images: MSR-Resolved gives the best FD-DINOv2, and MSR-Direct the best KID. KID is unbiased at small sample sizes, which makes it the more reliable of the two here. CMMD gives a more mixed picture: MSR-Direct is on par with the baselines, while MSR-Resolved scores worst. The uniform sampler, compared with MSR-Direct, is more diverse but has worse KID, worse CMMD, and the lowest validity of all methods.

3D chairs. We also computed reference-based metrics for the chairs, against the corpus of reference designs. We used two weightings (equal mass per design family and equal mass per design), and both image embeddings (DINOv2) and text embeddings of chair descriptions. Each chair was described by GPT-5.6 Sol, and the description was embedded with text-embedding-3-large:

Metric (equal family mass)MSRIndependentCoordinated
FD (image) ↓0.7600.7931.168
FD (text) ↓0.1680.2240.201
KID ↓0.0580.0720.063
CMMD ↓2.443.183.05
MMD, DINOv2 (RBF) ↓0.1450.1680.253
MMD, text (RBF) ↓0.0580.1650.099

MSR is closest to the reference distribution on every metric, and the same holds with equal per-design weighting (e.g., KID 0.058 vs. 0.069 and 0.066; CMMD 2.40 vs. 3.19 and 3.07). Notably, coordinated generation, which asks the agent itself to diversify, moves further from the reference than independent generation on image FD and MMD. MSR's diversity, by contrast, stays aligned with the corpus.

AuthorsOpenReview ↗

Response to Reviewer ueCB (2/4)

2. Reliability of the diversity evaluation

Vendi bandwidth. The base bandwidth is the median nonzero within-prompt Euclidean distance between L2-normalized features, pooled over all four methods. It is therefore a single value per encoder, shared by all methods, and not tuned for any one of them. We recomputed Vendi with bandwidths from 0.25× to 4× this value, for both encoders (mean over the 50 prompts; raw scores, bounded by 10):

EncoderBandwidth (× median)Prompt onlyIndependent rewriteCoordinated rewriteMSR-Resolved
CLIP0.259.139.739.9310.00
CLIP0.55.586.848.249.87
CLIP12.373.023.866.68
CLIP21.361.531.742.59
CLIP41.101.151.201.41
DINOv20.258.209.169.709.97
DINOv20.54.516.147.719.38
DINOv212.163.194.326.73
DINOv221.331.692.042.93
DINOv241.091.201.291.50

MSR-Resolved is the most diverse method at every bandwidth and with both encoders, and the full ranking of methods is identical across all settings. At the extremes, all scores compress toward 10 (very small bandwidth) or 1 (very large bandwidth), as expected, but the ordering is preserved.

To test this per prompt, the next table reports the paired difference in Vendi between MSR-Resolved and each baseline, with 95% bootstrap confidence intervals over prompts and the number of prompts on which MSR scores higher:

EncoderBandwidth (× median)MSR − Prompt onlyMSR − Independent rewriteMSR − Coordinated rewrite
CLIP0.250.87 [0.57, 1.22] (50/50)0.27 [0.16, 0.39] (50/50)0.07 [0.03, 0.12] (49/50)
CLIP0.54.28 [3.72, 4.85] (50/50)3.02 [2.54, 3.51] (50/50)1.62 [1.31, 1.95] (50/50)
CLIP14.30 [3.97, 4.64] (50/50)3.65 [3.30, 4.00] (50/50)2.82 [2.50, 3.13] (50/50)
CLIP21.23 [1.12, 1.35] (50/50)1.06 [0.95, 1.17] (50/50)0.84 [0.74, 0.95] (50/50)
CLIP40.31 [0.28, 0.33] (50/50)0.26 [0.23, 0.29] (50/50)0.20 [0.18, 0.23] (50/50)
DINOv20.251.78 [1.32, 2.25] (50/50)0.82 [0.51, 1.16] (49/50)0.28 [0.13, 0.46] (48/50)
DINOv20.54.87 [4.30, 5.40] (50/50)3.24 [2.69, 3.80] (49/50)1.67 [1.27, 2.09] (48/50)
DINOv214.57 [4.07, 5.05] (50/50)3.54 [3.09, 3.98] (50/50)2.42 [2.04, 2.79] (49/50)
DINOv221.60 [1.39, 1.81] (50/50)1.24 [1.06, 1.42] (49/50)0.89 [0.74, 1.04] (48/50)
DINOv240.41 [0.36, 0.46] (50/50)0.30 [0.26, 0.35] (49/50)0.21 [0.18, 0.25] (47/50)

All 30 confidence intervals exclude zero, and MSR scores higher on at least 47 of the 50 prompts in every comparison. We added both tables to the appendix.

Odd-one-out confound. The reviewer is right that pooling 30 baseline images with 10 MSR images could favor any distinct-looking group. Following the suggestion, we reran the evaluation as pairwise comparisons with equal pool sizes: 10 MSR-Resolved images vs. 10 images from one baseline, for each of the 50 prompts, with GPT Image 2. The judge repeatedly removes the most distinct image; removed images receive ranks 1–10 in order of removal, and the remaining images receive rank 20. A lower mean rank therefore means a more diverse set. To address the concern about judge independence, we report two judges: GPT-5.6 Sol, as in the paper, and Kimi K3, which was not involved in generation, rewriting, or resolution.

JudgeComparatorMSR mean rank ↓Comparator mean rankPaired gap [95% CI]Prompts with a lower MSR mean removal score
GPT-5.6 SolPrompt only5.5020.0014.50 [14.50, 14.50]50/50
GPT-5.6 SolIndependent rewrite5.9219.5813.67 [13.14, 14.10]50/50
GPT-5.6 SolCoordinated rewrite8.6916.818.12 [6.84, 9.30]45/50
Kimi K3Prompt only5.5519.9514.40 [14.24, 14.50]50/50
Kimi K3Independent rewrite5.9119.5913.67 [13.12, 14.13]50/50
Kimi K3Coordinated rewrite10.0515.455.40 [3.84, 6.88]45/50

With equal pools, MSR is judged more diverse than every baseline, by both judges, with confidence intervals well away from zero. Against prompt-only generation and independent rewriting, MSR has the lower mean removal score on all 50 prompts for both judges. Against prompt-only generation with GPT-5.6 Sol, every MSR image is removed before any baseline image on every prompt, which gives the maximum possible gap of 14.5. Against coordinated rewriting, the strongest baseline, MSR has the lower mean removal score on 45 of 50 prompts for both judges. The independent judge leads to the same conclusions as the original one.

AuthorsOpenReview ↗

Response to Reviewer ueCB (3/4)

3. Test-time diversity baselines

We thank the reviewer for these references, and we have added a discussion of these methods to the related-work section. There is an important difference in applicability. CADS, Particle Guidance, and SPARKE all modify the diffusion sampling process, so they require white-box access to the model's sampling process. They cannot be applied to proprietary image generators such as GPT Image 2 or MAI-Image-2.5, nor to non-diffusion generators such as LLMs or tool-using agents. MSR only changes the input to the generator, so it applies to any model, open or proprietary, whatever its architecture. This is what allows us to use the same method across text, webpages, images, and agent-generated 3D assets. The two approaches are also complementary. Guidance methods diversify the rendering of a single prompt, mostly through visual variation. MSR diversifies what is rendered, at the level of semantic attributes drawn from the corpus, and aligns it with the corpus distribution. When the model is open, MSR's sampler could be combined with a guidance method as the renderer.

4. Internal inconsistencies

We thank the reviewer for catching the inconsistency between Table 4 and Table 15. Table 15 was mistakenly taken from an earlier run. We have replaced it with the per-criterion breakdown of the run reported in the main text, so the two tables now agree. We have also clarified in App. B.6.4 and the caption of Table 15 that the judge chose and froze the five criteria, their weights, and scoring anchors before viewing any candidate. At that point it had received only the common design assignment and a neutral evaluation framing, not the corpus, generation discussions, sampled labels, group identities, or expected result.

5. Validity evaluation

We agree that validity rates for the baselines are needed to assess the quality cost of MSR. Validity (the image satisfies the request and is visually acceptable) is defined for any image, so we now report it for all methods, using the same judge and protocol as Table 14. Attribute realization, by contrast, measures whether the sampled attributes appear in the image, so it applies only to MSR.

GeneratorPrompt onlyIndependent rewriteCoordinated rewriteMSR-DirectMSR-Resolved
GPT Image 210099999997
MAI-Image-2.510095989697
FLUX.2-flex9998999590

For GPT Image 2 and MAI-Image-2.5, MSR's validity is within 1–4 points of the baselines (1–4 images out of 100), and on MAI-Image-2.5 both MSR variants are above independent rewriting. The exception is MSR-Resolved with FLUX.2-flex, where validity drops to 90%. Here, resolution trades some validity for attribute realization (95% → 90% validity, 71.1% → 77.9% attributes realized), so for this generator MSR-Direct (95%) is the better operating point when validity matters most. We hypothesize that FLUX.2-flex's text encoder struggles with the long, attribute-rich prompts that resolution produces. Together, these results suggest that the diversity MSR adds does not come at a high cost in realism. We have added this table and discussion to the appendix.

Beyond LLM judges, every central claim is also supported by metrics that involve no judge. Diversity is measured with embedding-based Vendi scores (Section 2), which rank MSR first at every bandwidth and with both encoders. Alignment with real data is measured with reference-based metrics (FD, KID, CMMD, and MMD; Section 1), which support the same conclusions, including for the 3D chairs.

AuthorsOpenReview ↗

Response to Reviewer ueCB (4/4)

6. Novelty

We agree that MSR's components are individually known. We see the contribution in the separation itself: identifying that the generator is a poor sampler, and showing that an explicit, interpretable, corpus-derived sampler fixes this across very different domains and for black-box models and agents. This differs from existing methods in three ways:

  • Attributes and statistics come from data, not from the LLM's prior. Attribute-guided methods such as AttrPrompt let an LLM propose attributes and sample them independently. MSR mines attributes from a reference corpus and samples from their estimated joint distribution. In a new comparison in user simulation, AttrPrompt, whose attributes are proposed by an LLM does not align well with real users (Table below).
  • It works with any generator. Unlike diversity-guidance methods (CADS, Particle Guidance, SPARKE), which modify diffusion sampling, MSR only changes the generator's input. It therefore applies equally to proprietary image models, LLMs, and tool-using agents, which is what makes a single method usable across all five domains.
  • Separation can mitigate the diversity-quality trade-off. Making the generator itself more diverse costs quality. For the 3D chairs (Table 4), coordinated prompting lowers quality from 82.35 to 70.80, while MSR is more diverse and keeps quality at 83.55; for text-to-image, our results suggest that the diversity MSR adds does not come at a high cost in realism. We compare with AttrPrompt, using GPT-5-mini as the LLM that proposes attributes as this was the model used to create the schema in MSR.
MethodForward KLJSCoverage
Prompt0.5180.5360.567
PersonaHub0.5220.5370.562
AttrPrompt (GPT-5-mini schema, uniform sampling)0.5540.5720.484
MSR fine-grained0.3060.3080.773
MSR compact0.3180.3190.747

7. Labeling accuracy and cost

Labeling accuracy. Outside the controlled weather setting, we also evaluated corpus labeling in a four-page Design2Code pilot using the 5,484-attribute schema. A separate GPT-5.5 audit checked the assignments produced by the GPT-5.6 labeling procedure and searched the full attribute menu for omissions. It yielded model-audited precision and recall proxies of 98.2% and 99.3%, respectively.

Cost. We logged the LLM requests and reconstructed the token usage needed to build the user-simulation schemas from the saved prompts and outputs:

SchemaLLM requests (GPT-5-mini)Input tokensOutput tokens (saved JSON)
Compact (84 attributes, 600 conversations)2,1837.46M1.62M
Fine-grained (465 attributes, same 600 conversations)2,31317.86M1.24M

Building a schema is a modest, one-time investment per reference corpus (~2,200–2,300 GPT-5-mini calls and <20M input/output tokens, a relatively small one-time token cost for an entire corpus, especially compared with generating thousands of downstream outputs). Once built, the schema and corpus statistics can be reused across many generations and different generators without repeating the mining and labeling stages, amortizing the upfront cost. The sampling cost afterward is negligible compared to the cost of generating the outputs.

We thank the reviewer again. We believe these additions, especially the reference-based metrics and the uniform-sampling comparison, substantially strengthen the alignment claim.

Reviewer_nTFDOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the important challenge of handling diversity when sampling from generative models. At a high-level, it uses a LLM to build a scheme of attributes and generate a matrix of frequencies and dependencies. Various configurations are then sampled using ancestral sampling and topic modeling which are then fed to the conditional generator for producing the desired output. Experiments are shown on a wide variety of settings including weather, webpages, text-to-image, and 3D design.

Overall: The paper solves an interesting problem in the literature and is well presented but some details on rare attributes and evaluation need further consideration.

Critical strengths

(a) The approach to enable diversity is elegant in that it separates the generating module from the diversity component by expliciting conditioning it via the schema approach. Also, it is human readable and can be potentially edited. (b) Experiments show the benefits of the approach in a wide variety of settings including weather where they show excellent coverage numbers. The KL and JS metrics reduce substantially for simulators while the quality stays high.

Critical weaknesses

(i) The main issue is that rare attributes can be lost in multiple stages of the process including when using thresholds and sampling according to their frequencies. It captures the bulk of the distribution but not necessarily the tail which might be relevant for stress testing scenarios. (ii) The theory behind preserving the joint distribution of the attributes is missing and needs to be understood in further detail. At the moment, it is experimentally driven and also the evaluation metrics need to be understood better in the sense that diversity may come at the cost of realism.

Questions for authors
  1. How can we handle rare attributes? Can we use prior distribution for the rare categories for instance?
  2. How does the approach compare to the vanilla baseline of just uniform sampling over the explicit attributes? Will this tradeoff coverage for diversity and maybe an evaluation of diversity, coverage, and realism would make this trade-off more explicit.
4 comments
AuthorsOpenReview ↗

Response to Reviewer nTFD (1/3)

Thank you for your time and thoughtful suggestions. We are glad you found our proposal to separate the diversity component from the generator elegant, and the human-readable, editable schema useful. Below we address each of your concerns.

The key observation behind MSR is that modern generative models are excellent renderers but poor samplers: asked to produce many outputs, they collapse onto a narrow set of modes, even when explicitly instructed to be diverse. MSR therefore takes sampling away from the generator. Diversity comes from an explicit sampler over an interpretable schema, and the generator only renders each sampled configuration.

The two samplers in the paper, ancestral sampling and topic-model sampling, are a starting point we chose, as the most natural default target is the corpus distribution. Any sampler over the schema is an instance of MSR: uniform sampling, tempered sampling, tail-weighted sampling, or a sampler hand-designed for stress testing. The mine and render stages, and the argument for why the separation works, stay exactly the same. We make this framing more explicit in the introduction.

In this sense, we view both scenarios in Q1 and Q2 as ones that MSR handles directly through the choice of sampler. To over-represent rare attributes, one can increase smoothing or use a sampler that specifically targets them. Where maximum diversity is required, even at the cost of adherence to the corpus, one can sample attributes uniformly. We view this as an explicit strength of MSR as a framework.

We address the three raised concerns below, which we consider connected, with additional experiments and, following your suggestion, a formal bound:

  • Rare-attribute coverage. The ancestral sampler already uses additive smoothing, which is equivalent to a Dirichlet prior, and a standard 10,000-configuration sample reaches every rare attribute in the WebSight schema.
  • Diversity vs. realism. We extend the paper's validity evaluation to all baselines and add reference-based metrics (FD-DINOv2, KID, CMMD) against held-out real images. MSR's diversity gains come with validity very close to the baselines' and with outputs closer to real images. We also add a uniform sampler, which shows the cost of moving away from the corpus: higher diversity, but lower validity in images and worse alignment and coverage in user simulation.
  • A theoretical analysis. Following the suggestion, we formalize why separating sampling from rendering preserves the target distribution. We show that the distance between the rendered attribute distribution and the target distribution is bounded by the sampler's error plus the renderer's error, both when requiring the entire attribute configuration to match exactly and when measuring the fraction of attributes matched (partial adherence). We also extend the bound to include the validity of outputs as a rendering error, beyond the realization of attributes.

W(i) / Q1: Rare attributes

We test whether the ancestral sampler used in the paper can preserve rare attributes at different rarity thresholds. We use the existing WebSight attribute matrix (10,000 pages ×\times 485 attributes) and the ancestral sampler from the paper with strong smoothing, as used for the WebSight experiments (β=1\beta=1, compared with β=0.2\beta=0.2 in user simulation; m = 10 in both). This additive smoothing is equivalent to placing a Dirichlet prior on each categorical distribution, which is exactly the kind of prior over rare categories the reviewer suggests in Q1.

Source-frequency cutoffEligible labelsSource pages with anySampled configurations with anyDistinct labels reached
<=0.1%88376/10,000 (3.76%)9,978/10,000 (99.78%)88/88
<=1%2664,797/10,000 (47.97%)10,000/10,000 (100%)266/266
<=5%3979,705/10,000 (97.05%)10,000/10,000 (100%)397/397

Ancestral sampling with strong smoothing (β=1\beta=1) was sufficient to cover every rare attribute in 10,000 samples, at all three cutoffs, and rare labels appear in far more sampled configurations than in the corpus (e.g., 99.78% vs. 3.76% at <=0.1%). In user simulation, we used a lower β\beta (less smoothing) to match the real distribution more closely. This shows that the sampling distribution can be tuned to the goal, and MSR inherits the large body of work on probabilistic modeling for doing so.

Finally, attributes that were missed during the mining stage, or are absent from the corpus entirely, can still be included. As noted, the schema is human-readable and editable, so such attributes can be added by hand and matched against the corpus at the backfill step. This lets users cover rare cases they already know matter, for example, for stress testing, even when the initial mining missed the attributes.

AuthorsOpenReview ↗

Response to Reviewer nTFD (2/3)

W(ii) / Q2: Diversity vs. realism and a uniform sampling baseline

We agree that diversity can come at the cost of realism, and that this trade-off deserves careful evaluation. In MSR, the two are handled by separate components: diversity is determined by the sampler, while realism is determined mainly by the renderer, which only has to produce a single high-quality output for each sampled configuration. Our quality evaluations support this:

  • Agentic 3D folding chairs (Sec. 3.5, Table 4). Asking the agent to make each chair different from the previous ones (coordinated generation) adds some variety but lowers quality from 82.35 to 70.80. MSR gives the most diverse set (mean removal rank 17.1, vs. 25.1 for coordinated and 48.8 for independent generation) while keeping quality at 83.55, slightly above independent generation. The per-criterion breakdown in App. B.6.4 shows MSR chairs scoring higher than coordinated ones on every rubric criterion, including structural coherence and the folding mechanism.
  • Text-to-image (Sec. 3.4). We extend the paper’s validity results (App. B.5.7, Table 14), which cover only MSR, by judging the baselines with the same judge and protocol (validity = the image satisfies the request and is visually acceptable; % of images):
GeneratorPrompt onlyIndependent rewriteCoordinated rewriteMSR-DirectMSR-Resolved
GPT Image 210099999997
MAI-Image-2.510095989697
FLUX.2-flex9998999590

MSR's validity stays at 95–99% in almost all settings, within a few points of the baselines, while increasing Vendi diversity well beyond prompt rewriting (Table 3). The exception is MSR-Resolved with FLUX.2-flex, where validity drops to 90%. We hypothesize that FLUX.2-flex's text encoder struggles with the long, attribute-rich prompts that resolution produces. Together, these results suggest that the diversity MSR adds does not come at a high cost in realism. The chair results also show the opposite case: asking the generator to diversify itself costs quality.

Uniform sampling. Following your suggestion, we added a uniform-sampling baseline, which samples attributes uniformly over the same mined schema, in two settings. Alongside diversity and validity, we now compare the generated outputs directly with reference data, which evaluates the full sampling-and-rendering pipeline without relying on the mined labels.

Text-to-image (GPT Image 2). We report reference-based metrics against held-out COCO images (500 images per method; validity on 100 images per method):

MethodVendi (CLIP) ↑Vendi (DINOv2) ↑FD-DINOv2 ↓KID ↓CMMD ↓Validity ↑
Prompt only2.372.160.9370.01630.860100.0%
Independent rewrite3.023.190.8630.01640.91599.0%
Coordinated rewrite3.864.320.7860.01120.83799.0%
MSR-Direct5.144.970.7190.00570.85499.0%
MSR-Resolved6.686.730.6390.00841.06097.0%
MSR-Direct + uniform sampler6.055.290.6970.01020.99296.0%

Both corpus-based MSR variants move outputs closer to real images than the prompt-based baselines on FD-DINOv2 and KID (with MSR-Resolved achieving the best FD-DINOv2 at 0.639 and MSR-Direct the best KID at 0.0057, while CMMD is more mixed). Compared with MSR-Direct, the uniform sampler increases diversity (Vendi 6.05 vs. 5.14 with CLIP and 5.29 vs. 4.97 with DINOv2) but worsens KID (0.0102 vs. 0.0057), worsens CMMD (0.992 vs. 0.854), and lowers validity (96.0% vs. 99.0%), the lowest of all methods. A uniform sampler deliberately flattens the attribute distribution, producing more combinations that rarely occur together in real images and are harder to render convincingly, whereas the corpus-based sampler keeps configurations plausible.

User simulation (GPT-5.5). Here a reference distribution is available, so we can measure the effect of the sampler on distributional alignment directly:

MethodForward KL ↓JS ↓Coverage ↑
Prompt only0.5180.5360.567
PersonaHub0.5220.5370.562
MSR compact, ancestral sampler0.3180.3190.747
MSR fine-grained, ancestral sampler0.3060.3080.773
MSR fine-grained, uniform sampler0.4590.4660.616

Uniform sampling still improves on prompt-only generation and PersonaHub, showing the value of the mined schema itself, but most of MSR's alignment gain comes from sampling according to the corpus statistics. Notably, uniform sampling reduces coverage even though it spreads mass more evenly over the schema: many of its configurations are combinations that real users rarely exhibit, so they fall outside the real behavior clusters instead of covering more of them.

AuthorsOpenReview ↗

Response to Reviewer nTFD (3/3)

Together, the image and user-simulation results give a consistent picture of the trade-off: uniform sampling buys diversity at the cost of validity and alignment, while corpus-based sampling increases diversity in a way that stays faithful to the real distribution, but crucially, both are available within MSR through the choice of sampler alone.

W(ii): Theory behind preserving the joint distribution

We agree that the paper would benefit from a theoretical analysis of how MSR preserves the joint attribute distribution. We add a formal proposition showing that the attributes of the rendered outputs match the target distribution whenever (1) the sampler matches the target distribution and (2) the renderer faithfully realizes the sampled attributes. In other words, once sampling is separated from the generator, the generator's own mode collapse no longer governs the attribute distribution; only the sampler's accuracy and the renderer's instruction-following do.

Exact configuration matching. Let p∗p^{\ast} be the target distribution over attribute configurations (the corpus distribution for our default samplers, or a uniform or tail-weighted distribution when those are the goal). Let qq be the sampler's distribution, and p^\hat{p} be the distribution of attribute configurations a(x)a(x) in the rendered outputs xx. Coupling each sampled configuration c∼qc \sim q with its rendering xx gives:

TV(p^,p∗)≤TV(q,p∗)+Pr⁡[a(x)≠c],\mathrm{TV}(\hat{p}, p^{\ast}) \le \mathrm{TV}(q, p^{\ast}) + \Pr[a(x) \ne c],

where the first term is the sampler error and the second is the renderer error.

The distributional gap splits cleanly into two terms:

  • Sampler error, TV(q,p∗)\mathrm{TV}(q, p^{\ast}), which measures how well the statistical sampler matches the chosen target p∗p^{\ast}. For any sampling order π\pi with joint distribution qπq_\pi, Pinsker's inequality and the chain rule for KL divergence decompose this error across the conditional steps: TV(qπ,p∗)≤12KL(p∗∥qπ),KL(p∗∥qπ)=∑iEp∗[KL(p∗(cπ(i)∣cπ(<i)) ∥ qπ(cπ(i)∣cπ(<i)))]\mathrm{TV}(q_{\pi}, p^{\ast}) \le \sqrt{\frac{1}{2}\mathrm{KL}(p^{\ast} \Vert q_{\pi})}, \qquad \mathrm{KL}(p^{\ast} \Vert q_{\pi}) = \sum_{i} \mathop{\mathbb{E}}\nolimits_{p^{\ast}} \Big[ \mathrm{KL}\Big( p^{\ast}(c_{\pi(i)} \mid c_{\pi(\lt i)}) \,\Vert\, q_{\pi}(c_{\pi(i)} \mid c_{\pi(\lt i)}) \Big) \Big] where the conditionals qπ(cπi∣cπ<i)q_\pi(c_{\pi_i} \mid c_{\pi_{<i}}) include our smoothing, backoff, and compatibility mask. Because the sampler averages over random orders π\pi, convexity gives TV(q,p∗)≤Eπ[TV(qπ,p∗)]\mathrm{TV}(q, p^{\ast}) \le \mathop{\mathbb{E}}\nolimits_{\pi}\big[\mathrm{TV}(q_{\pi}, p^{\ast})\big]. Thus, accurate per-step conditionals yield a small sampler error.

  • Renderer error, Pr⁡[a(x)≠c]\Pr[a(x) \ne c], which is how often the rendered output fails to match the full sampled configuration cc.

Partial attribute matching. In the total-variation bound above, Pr⁡[a(x)≠c]\Pr[a(x) \ne c] is all-or-nothing: when a schema has DD attributes, an output that realizes 19 of 20 sampled attributes and misses one is penalized the same as an output that misses all 20. If we instead measure the fraction of mismatched attributes using normalized Hamming distance, dH(a,c)=1D∑i=1D1{ai≠ci}d_H(a, c) = \frac{1}{D}\sum_{i=1}^{D} \mathbf{1}\{a_i \ne c_i\}, the exact same coupling argument bounds the corresponding Wasserstein distance WHW_H:

WH(p^,p∗)≤WH(q,p∗)+E[dH(a(x),c)].W_H(\hat{p}, p^{\ast}) \le W_H(q, p^{\ast}) + \mathbb{E}\big[d_H(a(x), c)\big].

Here the renderer error E[dH(a(x),c)]\mathbb{E}[d_H(a(x), c)] is simply the average fraction of attributes that the renderer fails to match (whether by omitting a requested attribute or adding an unrequested one). And because WH(q,p∗)≤TV(q,p∗)W_H(q, p^{\ast}) \le \mathrm{TV}(q, p^{\ast}), the same KL bound on the sampler error still applies directly.

Including output validity. Realizing the requested attributes and producing a valid, realistic output are both required in practice. Both bounds extend directly to the joint distribution of attributes and validity: in the exact bound, the renderer error becomes the probability of "an attribute mismatch or an invalid output," and in the partial-adherence bound, the probability of an invalid output is simply added to the average fraction of mismatched attributes.

Together, this analysis connects the two parts of our response: the sampler qq controls which attribute distribution we target (whether matching the corpus or boosting rare attributes via smoothing), while adherence and validity measure how well the renderer realizes those samples. Both proofs are short (triangle inequality plus coupling) and are added to the appendix.

We thank you again for the thoughtful suggestions, which helped us sharpen the framing of MSR as a framework and add the theoretical analysis and experiments above.

Thanks to the authors for coming back with strong responses for all 3 questions. I think the idea is simple but has been effectively demonstrated to generate diverse samples and a bit of theory has been added to showcase that the joint distribution can be potentially preserved under certain conditions and considering both the rare attributes as well as diversity/realism trade-off. I will maintain my score in lieu of the limited novelty as has been noted by other reviewers as well.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.