acceptodds
Under review as a conference paper at ICLR 2027

Where You Inject Diversity Matters: A Unified Framework for Diverse Generation

Abstract

Open-ended generation tasks often require a set of meaningfully different outputs, yet large language models often produce similar generations. Existing test-time diversity methods operate at different stages of generation with varying effectiveness, but it remains unclear what design choices lead to meaningful diversity in the output. We introduce a framework that characterizes test-time diverse generation methods by the diversity source introduced during generation and provide a transmission score that quantifies how effectively variation in the source is realized in the final output. Guided by this framework, we instantiate two fully automated methods that first generate diverse intermediate specifications and then condition on them to produce final responses. Across five open-ended tasks and four backbone models, specification-level injection improves output diversity over test-time baselines while maintaining comparable quality. Our analysis shows that successful diversity injection depends on both the diversity of the sources and their transmission to the output, highlighting source design and source-to-output realization as two key levers for building more diverse generation systems.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Output homogenization, diversity, inference-time methods

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CmzEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper posed a framework (Diversity Injection Framework) that groups methods of improving model diversity at inference-time into three levels, based on what the output is conditioned on: nothing (level 0), a source with no semantic content (level 1, eg. a random string), and specification of the output (level 2). It also defined a transmission score for measuring how much of the variation in the source ends up in the output. It then proposed two level 2 methods (Diverse Outline and Keyword) that improve model diversity by generating diverse output specifications before getting to the final output. Both achieved higher diversity than existing methods across 5 open-ended tasks.

Critical strengths
  1. The proposed methods are simple and the improvement on model diversity is consistent. Across 5 open-ended tasks, both methods get better results than baselines, while the quality is comparable.
Critical weaknesses
  1. I think the novelty and contribution of the framework and two methods are narrower than claimed. Section 3.2 said “there is no existing test-time method provides per-output specifications to promote diversity in general open-ended generation”. Intent Factored Generation (https://arxiv.org/pdf/2506.09659) is close to what the paper described as a level 2 method.
  2. The transmission score analysis setup has some issues. (1) The analysis only compares with a single baseline SSOT, since the author found no existing level 2 method. However, another level 1 baseline (Random Concept) is not evaluated. (2) The comparison with SSOT itself is also not meaningful. SSOT is designed to use a random string as a source of randomness. It doesn’t need to have the string recovered from the output. So the transmission score is ~0, which doesn't show the source is ignored or level 2 is better.
Questions for authors
  1. The paper’s title asks “where you inject diversity matters”. Could the author elaborate on this idea? The paper reads more about “what diversity is injected”, as the main contribution (the framework) categorizes methods based on the specification of output in the source.
4 comments
AuthorsOpenReview ↗

Part 1 of 3: novelty and Intent Factored Generation (W1)

We thank the reviewer for highlighting Intent Factored Generation (IFG). We would like to clarify that our primary contribution is not the use of intermediate specifications itself, but a unified framework for characterizing diversity injection, together with a transmission-based analysis of how variation in diversity sources is reflected in final outputs. Our work systematically investigates the relationship between specification diversity and output diversity through controlled experiments, providing insights that might guide the design of future diverse generation methods. IFG can be naturally categorized as a Level 2 method under our framework. IFG samples semantic intents independently for individual responses, while Diverse Outline and Keyword diversify a set of intermediate specifications through joint generation. We additionally evaluated IFG on Qwen3-4B using NoveltyBench, with 100 prompts and 20 outputs per prompt. We followed the authors' few-shot prompting procedure, adapting the examples to our tasks, and tested two intent/response temperature configurations: (1.0, 0.5) and (1.2, 0.7).

MethodLLM DivEmb DivStruct DivDistinct-N
Direct2.720.300.1810.96
(a) IFG as released (1.0, 0.5)3.270.400.1711.00
(a) IFG as released (1.2, 0.7)3.680.440.1812.99
(b) Independent specifications (1.0, 1.0)2.960.260.157.33
(b) Independent specifications (1.2, 0.7)3.120.280.167.48
Keyword5.500.380.2413.21
Diverse Outline5.540.410.2713.85

For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time.

Keyword and Diverse Outline achieve substantially higher LLM-judged and structural diversity than IFG in this experiment, while IFG achieves higher embedding diversity at the higher-temperature configuration. These results further support the effectiveness of constructing diverse specifications at the output-set level.

AuthorsOpenReview ↗

Part 2 of 3: the transmission analysis (W2)

We thank the reviewer for these comments. We would like to clarify that the transmission score is intended as a diagnostic measure of the relationship between injected sources and final outputs. Specifically, it quantifies the fraction of source information that is recoverable from the output. A low transmission score does not necessarily imply that the source is ignored. Rather, it indicates that only a small fraction of the source information is recoverable from the output, although the source may still influence the generation process. Additional Random Concept baseline. We evaluated Random Concept (Agrawal & Goyal, 2026) on Qwen3-4B and GPT-5-mini across Infinite Chats and NoveltyBench. Following its random-word injection strategy, we prepend a randomly sampled word from the wonderwords package used by the paper to each task and generate 20 responses per prompt. The results are:

Backbone, datasetMethodLLM DivEmb Div
Qwen3-4B, Infinite ChatsDirect2.320.19
Qwen3-4B, Infinite ChatsRandom Concept2.480.15
Qwen3-4B, Infinite ChatsDiverse Outline6.440.35
Qwen3-4B, NoveltyBenchDirect2.720.30
Qwen3-4B, NoveltyBenchRandom Concept2.660.22
Qwen3-4B, NoveltyBenchDiverse Outline5.540.41
GPT-5-mini, Infinite ChatsDirect3.400.26
GPT-5-mini, Infinite ChatsRandom Concept2.920.17
GPT-5-mini, Infinite ChatsDiverse Outline8.010.40
GPT-5-mini, NoveltyBenchDirect3.920.41
GPT-5-mini, NoveltyBenchRandom Concept3.310.24
GPT-5-mini, NoveltyBenchDiverse Outline8.330.50
Random Concept does not consistently improve diversity over Direct Generation in these settings, whereas Diverse Outline achieves substantially higher diversity. Embedding diversity shows a similar pattern.

For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time.

Using the revised estimation procedure described in our response to HWs2, we obtain the following transmission scores on Qwen3-4B:

MethodInfinite ChatsNoveltyBench
Diverse Outline0.820.89
Random Concept0.370.47
SSoT0.0020.014

These results illustrate different degrees of recoverability of source information in the final outputs, with Diverse Outline achieving higher estimated transmission than Random Concept and SSoT. Importantly, the transmission score complements rather than replaces direct measures of output diversity.

AuthorsOpenReview ↗

Part 3 of 3: "where" versus "what" (Question)

Thank you for raising this interesting question. By “where diversity is injected,” we refer to the point in the generation process at which variation is introduced to encourage different outputs. We also recognize that “what information is injected” is another valid perspective on our framework. In fact, we view “where” and “what” as closely related: the stage at which diversity is introduced naturally influences the type of information that can be injected and how it shapes the final outputs.

Reviewer_CmzEOpenReview ↗

Response to authors' rebuttal

Thank you for the detailed responses and additional experiments/analysis! I appreciate it and it did help strengthen the empirical comparison. However, I still think the contribution is incremental. I will keep my score.

Reviewer_wEAfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses response homogeneity in open-ended text generation. It proposes a three-level framework based on the source of diversity injection and defines a transmission score using conditional mutual information. The authors introduce two methods, Diverse Outline and Keyword, that provide distinct specifications for each response by jointly generating diverse outlines or combining values across multiple dimensions, respectively. Experiments across five datasets and four models show that both methods generally improve output diversity while maintaining comparable average quality scores. Ablation and intervention studies further examine the role of variation in specifications.

Critical strengths
  1. The methods are simple, practical, and demonstrate diversity gains across a broad evaluation. Both methods require no training and automatically generate intermediate specifications for different open-ended tasks. Results across five datasets and four models generally support their effectiveness, with gains in LLM-judged, embedding, and structural diversity. For example, on Infinite Chats with GPT-5-mini, Diverse Outline improves LLM-judged diversity from 2.69 to 7.32 and embedding diversity from 0.26 to 0.40 over Direct, while mean quality remains similar (8.30 for Direct versus 8.28 for Diverse Outline). These results suggest practical value.
  2. The paper uses interventions and controlled comparisons to investigate the source of the gains. Outputs generated from different specifications generally differ more than repeated samples from the same specification. The shared-axis intervention reduces specification diversity and observes a corresponding decline in output diversity (Figure 4). The paper also includes a two-step control (Tables 10–11). These analyses, particularly the shared-axis intervention, provide evidence that variation in specifications contributes to output diversity.
Critical weaknesses
  1. The experiments do not fully disentangle the contribution of diverse specifications from additional generation costs. The two-step ablation omits the control prompts, intermediate outputs, and details of matched conditions (Appendix B.3, Tables 10–11). The main experiments also report neither total token budget matching nor actual token usage by method (Section 5.1, Appendix A). Thus, while the results support the methods’ effectiveness, the source of the gains and their persistence under equal budgets remain insufficiently established.
  2. The robustness of Keyword’s fixed hyperparameters is not established. The method uses four axes with eight candidate values each across all tasks (Appendix C.2, Figure 9), without explaining these choices or analyzing sensitivity to either parameter. Since the scope for meaningful variation may differ across tasks, this omission limits confidence in the robustness of the fixed configuration.
Questions for authors
  1. What are the exact prompts and intermediate outputs for the 2 Step-only control in Tables 10–11? Does it use the same final-generation prompt as Diverse Outline, with matched intermediate text lengths? Do the main gains persist when only the instruction to make outlines distinct is removed?
  2. What is the total input and output token usage of each method in Tables 2 and 4–6? Do the proposed methods retain their diversity advantage under the same total token budget?
  3. Why does Keyword use four axes with eight candidate values each (Appendix C.2)? How sensitive are the results to these choices, and do fixed counts produce semantically redundant or unsuitable candidates for some tasks?
4 comments
AuthorsOpenReview ↗

Part 1 of 3: the 2 Step-only control (W1, Q1)

Thank you very much for your review and feedback!

The following are the details for 2 Step-only control experiments.

Prompts. Step 1 is one independent call per output with the user message Task: {task} and this system prompt:

You are a planning assistant.
Write a brief plan for how you will address the user's task: a compact list of
4-6 short keywords or phrases sketching your approach.

Output JSON only:
{"keywords": ["keyword1", "keyword2", "keyword3", "keyword4"]}

Step 2 is the output prompt of Diverse Outline (Figure 8), with the plan in the place of the outline. The final-generation prompt is therefore the same.

Intermediate outputs. NoveltyBench, "Name one reputed job." Example of the control's plan:

["clarify 'reputed' meaning", "select high-status occupation", "name one job", "give brief justification", "offer alternatives/follow-up"]

Diverse Outline compared with 2 step-only control: a first call proposes task-specific dimensions (the call of Figure 9), and a second call writes all 20 outlines, one value per dimension, under the rules "produce N DISTINCT outlines" and "no two outlines should share the same value for more than 1 axis". Figure 7 will show this prompt.

Example of Diverse Outline's 20 outline for the same prompt:

{"tone": "formal authoritative", "specificity": "single-word job title", "cultural_context": "US corporate culture", "presentation_style": "single-word answer only"}

Intermediate lengths. They are basically matched. Over 2,000 intermediates per dataset, the control's plans have 30.6 (Infinite Chats) and 29.5 (NoveltyBench) content tokens, and Diverse Outline's outlines 37.3 and 38.4. Total cost is close: 30,018 and 29,352 tokens per prompt for the control, against 32,756 and 34,509 for Diverse Outline.

Additional ablations. To further isolate the source of diversity gains, we evaluate two variants: (1) removing explicit distinctness instructions while retaining joint outline generation, and (2) generating each outline independently. All variants use the same final-generation prompt. The following table reports LLM-judged diversity (IC = Infinite Chats; NB = NoveltyBench):

MetricMethodQwen3-4B ICQwen3-4B NBGPT-5-mini ICGPT-5-mini NB
LLM DivDirect2.322.723.403.92
LLM DivOutlines written independently3.002.963.443.81
LLM DivOne call, distinctness instructions removed5.995.137.808.03
LLM DivDiverse Outline6.445.548.018.33
Emb DivDirect0.190.300.260.41
Emb DivOutlines written independently0.210.260.240.31
Emb DivOne call, distinctness instructions removed0.330.390.390.49
Emb DivDiverse Outline0.350.410.400.50
Struct DivDirect0.120.180.150.19
Struct DivOutlines written independently0.140.150.160.18
Struct DivOne call, distinctness instructions removed0.220.250.270.31
Struct DivDiverse Outline0.230.270.270.32

For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time.

We also report NoveltyBench’s Distinct-N in the same order: 10.96, 7.33, 13.12, 13.85 (Qwen3-4B) and 15.25, 8.85, 16.39, 16.75 (GPT-5-mini).

Removing the distinctness instructions preserves approximately 85%–96% of Diverse Outline's improvement over Direct Generation, whereas independently generated outlines achieve diversity close to Direct Generation. Embedding and structural diversity exhibit similar patterns. These results suggest that joint specification generation plays a more important role than explicit distinctness instructions alone. Together with the existing two-step control, they support our conclusion that the gains are not simply attributable to an additional generation step.

AuthorsOpenReview ↗

Part 2 of 3: token usage and equal budgets (W1, Q2)

We thank the reviewer for raising this question. Methods that inject diversity through intermediate specifications naturally incur additional token costs, as they require generating specifications before producing final outputs. We recorded the input and output tokens across all generation calls, excluding evaluation costs. Across 20 backbone–dataset configurations, Diverse Outline uses 5.0–7.3× the tokens of Direct Generation on GPT-5-mini, 2.2–3.2× on DeepSeek-V3.2, 2.4–3.4× on Mistral-Large-3, and 0.7–3.0× on Qwen3-4B.

Equal-budget comparison. To investigate whether the diversity improvements are simply due to additional token usage, we conducted controlled experiments on GPT-5-mini and Qwen3-4B across Infinite Chats (IC) and NoveltyBench (NB). We allocated Direct Generation and Verbalized Sampling budgets of BB and 2B2B, where BB is the average token usage of Diverse Outline for the corresponding setting.

Each baseline generates additional candidates until reaching the target budget, after which we select 20 outputs using greedy farthest-point selection based on embedding distance. This provides a strong comparison favoring the baselines, as the selection procedure explicitly maximizes embedding diversity. All outputs are evaluated using the same judge. The LLM-judged diversity results are:

CellMetricDirect, saved runDirect at BDirect at 2BVerbalized, saved runVerbalized at BVerbalized at 2BDiverse Outline
GPT-5-mini ICLLM Div2.083.203.332.532.783.006.45
GPT-5-mini ICStruct Div0.180.210.220.170.180.190.29
GPT-5-mini NBLLM Div2.433.253.493.103.043.106.52
GPT-5-mini NBStruct Div0.230.260.270.220.240.240.34
Qwen3-4B ICLLM Div1.441.852.071.861.832.024.71
Qwen3-4B ICStruct Div0.150.160.170.150.160.170.25
Qwen3-4B NBLLM Div1.762.062.272.542.512.574.13
Qwen3-4B NBStruct Div0.200.210.230.200.210.220.29

Diverse Outline consistently achieves substantially higher LLM-judged diversity than both baselines, even when they are allocated twice its token budget. At 2B2B, the improvements range from 1.56 to 3.49 points. Structural diversity exhibits the same pattern, with Diverse Outline outperforming both baselines by 0.06–0.10.

For embedding diversity and Distinct-N, the advantage is less consistent: the baselines match or exceed Diverse Outline in some settings, particularly when optimizing embedding-based selection.

These results demonstrate that the gains in LLM-judged and structural diversity cannot be explained by additional token usage alone.

AuthorsOpenReview ↗

Supplement to Part 2: input and output tokens for every row of Tables 2 and 4–6

Mean per prompt, in thousands of tokens, for all generation calls that produce the 20 outputs of a prompt (evaluation calls excluded). Counts are the usage logged at run time, except G2, which logs none and was recounted with the tokenizer. Keyword and Diverse Outline include the call that proposes the dimensions. For GPT-5-mini the output count includes hidden reasoning tokens. IC = Infinite Chats, NB = NoveltyBench, WP = WritingPrompts; "in" = input tokens, "out" = output tokens.

BackboneMethodIC inIC outNB inNB outWP inWP outPoetry inPoetry outArgKP inArgKP out
GPT-5-miniDirect0.24.90.25.30.25.50.25.90.34.7
GPT-5-miniVerbalized0.77.70.68.00.78.10.68.40.67.2
GPT-5-miniSSOT6.731.76.629.26.729.56.730.66.625.1
GPT-5-miniKeyword5.023.15.237.05.318.85.350.15.127.7
GPT-5-miniDiverse Outline5.327.45.229.35.623.15.238.95.221.5
Qwen3-4BDirect0.77.81.010.60.56.91.715.60.23.7
Qwen3-4BDirect, τ = 1.51.815.11.816.72.517.02.418.61.511.8
Qwen3-4BVerbalized0.82.70.82.70.83.70.84.50.94.2
Qwen3-4BG2109.45.5111.15.6110.45.5122.86.2100.65.0
Qwen3-4BSSOT4.86.74.76.65.08.64.810.94.76.7
Qwen3-4BKeyword5.14.75.05.05.45.45.15.45.15.3
Qwen3-4BDiverse Outline4.95.64.96.05.57.15.17.15.06.7
DeepSeek-V3.2Direct0.13.80.24.20.55.20.35.20.13.6
DeepSeek-V3.2Direct, τ = 1.50.84.21.14.31.24.70.64.90.33.9
DeepSeek-V3.2Verbalized0.62.40.52.41.04.00.63.60.53.4
DeepSeek-V3.2SSOT6.619.46.527.76.619.36.618.76.515.9
DeepSeek-V3.2Keyword5.53.45.24.15.75.95.24.35.24.7
DeepSeek-V3.2Diverse Outline5.55.15.25.75.77.95.46.55.26.8
Mistral-Large-3Direct0.13.30.33.60.24.50.24.90.43.7
Mistral-Large-3Verbalized0.73.20.73.01.04.90.64.40.74.8
Mistral-Large-3SSOT6.710.76.612.16.612.76.712.16.713.5
Mistral-Large-3Keyword5.24.95.25.85.76.75.34.75.36.3
Mistral-Large-3Diverse Outline5.16.45.27.25.78.75.36.75.48.5
AuthorsOpenReview ↗

Part 3 of 3: four dimensions × eight values (W2, Q3)

The configuration of 4 dimensions and 8 values per dimension was chosen as an untuned default. To examine its sensitivity, we conducted additional experiments on Qwen3-4B across Infinite Chats (IC) and NoveltyBench (NB), varying the number of dimensions A∈2,3,4,6A\in{2,3,4,6} and values per dimension V∈8,12V\in{8,12}, while keeping 20 outputs per prompt.

Infinite Chats:

Configuration: A × VLLM DivEmb DivStruct DivQuality
2 × 85.540.320.246.18
2 × 125.940.330.236.16
3 × 86.010.330.236.12
3 × 126.210.340.236.07
4 × 8 (default)5.900.330.226.07
4 × 126.410.350.225.98
6 × 85.840.340.195.95
6 × 126.210.350.205.86
Direct2.320.190.125.90

NoveltyBench:

Configuration: A × VLLM DivEmb DivStruct DivQuality
2 × 84.990.360.266.06
2 × 125.320.370.266.06
3 × 85.170.380.265.89
3 × 125.780.380.255.91
4 × 8 (default)5.500.380.245.80
4 × 125.540.380.245.87
6 × 85.500.370.235.77
6 × 125.370.380.225.74
Direct2.720.300.186.01

For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time.

How sensitive are the results? Only mildly. Every configuration has an LLM Div of 5.0–6.4, against 2.3–2.7 for Direct, and Emb Div varies by at most 0.04. Quality declines mildly as the grid grows, from 6.18 to 5.86 on Infinite Chats and from 6.06 to 5.74 on NoveltyBench.

Do fixed counts produce redundant or unsuitable candidates? Within a dimension, 1.5% to 2.7% of the value pairs are judged "essentially the same choice" for the task (every pair judged once by GPT-5.3-Chat).

These findings indicate that the method is not particularly sensitive to the exact configuration and that 4 × 8 is not necessarily optimal.

Reviewer_HWs2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies output diversity in open-ended LLM generation. It groups existing methods according to whether they use no explicit diversity source, an arbitrary source such as a random string, or a meaningful specification of the desired output. It proposes two prompting pipelines: Diverse Outline generates distinct keyword plans jointly, and the second one - Keyword generates dimensions and values and selects combinations using Hamming distance. The paper also defines an information-theoretic transmission score. Experiments cover five datasets and four backbone models, and they show improved diversity with generally comparable quality scores.

Critical strengths

1 - The empirical evaluation in this work covers several tasks and models, and the reported diversity improvements are substantial in several settings. Results across lexical, embedding-based, structural, and LLM-judged measures make the practical finding more informative than an improvement on a single metric.

2 - The source-variation experiments are useful. Comparing within-outline and between-outline diversity, and progressively fixing more keyword dimensions, provides evidence that the intermediate specifications contribute to output variation. Separating source diversity from the generator’s responsiveness to that source is also a potentially useful diagnostic perspective.

Critical weaknesses

1 - The methodological contribution appears limited relative to closely related work. In particular, Luminate (CHI 2024) already generates dimensions and values, constructs requirements by selecting values, and generates responses conditioned on those requirements. Although the authors cite Luminate and distinguishes its interactive setting, the underlying generation procedure is already automated. For Keyword, the clearest difference seems to be the Hamming-distance selection rule. However, a comparison against random selection with the same generated dimensions is not provided. I am not convinced that the experiments isolate the benefit of the proposed procedure over the closest existing approach. The broader proposition that varying meaningful conditioning instructions can improve output diversity is also well established.

2 - The transmission estimator does not appear to match the quantity defined in the theory. Equations 1–3 use sequence-level entropies, but Algorithm 1 normalizes output log probabilities by output length and source log probabilities by source length. Even with fixed lengths and exact probability estimates, this introduces a source-length/output-length factor into the resulting ratio. It is unclear how the reported values carry the interpretation of a fraction of source information transmitted. This is a critical weakness because the transmission analysis is presented as a distinct contribution beyond the prompting procedures.

Questions for authors

1 - Could you clarify the methodological difference from Luminate’s automated generation pipeline? A controlled comparison using the same axes and values, with random combination selection versus your Hamming-distance selection, would help establish your contribution.

2 - Can you explain the separate length normalizations in Algorithm 1 with the sequence-level definition of the transmission score? Are the reported scores computed exactly as the algorithm describes? Could you also show their sensitivity to the estimation-set size?

3 - Does the quality evaluator receive the original task? Figure 11 shows only an output ID and text, which is not sufficient to judge task adherence. Please also clarify how factual correctness is evaluated for factual tasks.

4 - Could you provide the exact Verbalized Sampling and SSoT prompts and relevant generation settings? Does the SSoT implementation include manipulating the random string to guide stochastic decisions, as specified in the original method?

6 comments
AuthorsOpenReview ↗

Part 1 of 4: Luminate and the selection rule (W1, Q1)

We thank the reviewer for highlighting the connection to Luminate. Luminate provides an important precedent for automatically generating dimensions and values and using their combinations to condition response generation. Its primary objective is to support interactive design-space exploration in human-AI co-creation. Our work instead studies diversity across a set of automatically generated outputs and systematically investigates how variation in intermediate specifications contributes to final-output diversity. To isolate the effect of the selection rule, we conducted the requested comparison using identical generated dimensions and candidate values, the same final-generation prompt, and 20 outputs per prompt across 100 prompts. Random selection independently samples one value per dimension, following Luminate's combination strategy, whereas Hamming uses our greedy max-min selection.

Backbone, datasetSelectionLLM DivEmb DivStruct DivQuality
Qwen3-4B, Infinite ChatsHamming5.900.330.226.07
Qwen3-4B, Infinite ChatsRandom5.900.330.226.08
Qwen3-4B, NoveltyBenchHamming5.500.380.245.80
Qwen3-4B, NoveltyBenchRandom5.330.380.245.82
GPT-5-mini, NoveltyBenchHamming7.710.430.296.42
GPT-5-mini, NoveltyBenchRandom7.760.440.306.34

For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time. The two selection rules yield comparable diversity and quality, suggesting that Hamming-distance maximization is not the primary driver of Keyword's effectiveness. Notably, with four dimensions and eight candidate values per dimension, two randomly sampled specifications already differ in 3.5 dimensions on average. Thus, random combinations already provide substantial specification variation, and further maximizing combinatorial distance does not necessarily improve output diversity. Clarification of our primary contribution. Our contribution goes beyond the use of outlines, keywords, or a particular selection heuristic. We introduce a unified framework for characterizing diversity injection, together with an information-theoretic analysis of how source variation is reflected in final outputs. Under this framework, Luminate, Keyword, and Diverse Outline can all be viewed as instances of specification-level injection, despite their different objectives and specification-construction strategies. Importantly, our work systematically investigates the relationship between specification diversity and output diversity through within- versus between-specification comparisons, controlled interventions on specification variation, and two-step ablations (Figures 2 and 4, Tables 10–12). These analyses provide evidence that varying meaningful specifications contributes to final-output diversity, while also revealing that maximizing source-level combinatorial distance does not always yield additional improvements. We will clarify the relationship to Luminate, incorporate the new controlled comparison, and further emphasize the framework and systematic analysis as our primary contributions.

AuthorsOpenReview ↗

Part 2 of 4: the transmission score (W2, Q2)

We thank the reviewer for this observation. We have revised the estimation procedure to better align with Definition 1 and recomputed the transmission scores accordingly. Specifically, we now compute all entropy quantities using whole-sequence log probabilities without length normalization. We also refine the marginal probability estimation by incorporating the generating source into the mixture, as the original held-out approximation may underestimate the marginal probability and consequently overestimate mutual information, particularly for small estimation sets. The revised marginal probability estimator is: P^(yℓ∣x)=pℓP(yℓ∣x,zℓ)+(1−pℓ)1M∑k∈EP(yℓ∣x,zk),\hat{P}(y_\ell\mid x) = p_\ell P(y_\ell\mid x,z_\ell) + (1-p_\ell)\frac{1}{M}\sum_{k\in E}P(y_\ell\mid x,z_k),

where pℓ=P(zℓ∣x)p_\ell=P(z_\ell\mid x), zℓz_\ell is the generating source, and (E) is the estimation set of size (M). This approximation incorporates the generating source explicitly and bounds the estimated per-sample information contribution by −log⁡2pℓ-\log_2 p_\ell.

Sensitivity to estimation-set size. We evaluate M∈{2,5,10,19}M\in\lbrace 2,5,10,19\rbrace, averaging over 20 random splits. IC and NB denote Infinite Chats and NoveltyBench, respectively. The revised transmission scores on Qwen3-4B are:

SourceIC, M=2M=2IC, M=5M=5IC, M=10M=10IC, M=19M=19NB, M=2M=2NB, M=5M=5NB, M=10M=10NB, M=19M=19
Diverse Outline0.900.860.820.790.940.910.890.87
Keyword0.910.860.820.780.960.930.900.88
SSOT0.0060.0030.0020.0010.0240.0170.0140.010

While the estimated scores decrease as (M) increases, the relative ordering remains consistent across all evaluated estimation set sizes. In particular, Diverse Outline and Keyword consistently achieve higher estimated transmission than SSoT. We will update Algorithm 1 and Tables 3 and 9 with the revised results and clarify the estimation procedure and its finite-sample sensitivity in the manuscript.

AuthorsOpenReview ↗

Part 3 of 4: the quality evaluator (Q3)

We have updated the quality evaluation prompt in Figure 11 to include the original task, with the input now formatted as {"id": N, "task": "...", "text": "..."}. All other evaluation instructions remain unchanged.All 211,536 outputs behind Tables 2 and 4–6 (119 method × backbone × dataset rows) were re-scored by GPT-6-luna. Quality is the mean of the four dimensions of Figure 11.

The updated quality scores for Table 2 are summarized below (IC = Infinite Chats, NB = NoveltyBench):

MethodGPT-5-mini ICGPT-5-mini NBQwen3-4B ICQwen3-4B NB
Direct6.907.005.906.01
Keyword7.086.426.075.80
Diverse Outline6.816.716.215.94

For all 20 backbone × dataset, we further compare the quality of our methods against Direct Generation using 95% paired bootstrap confidence intervals over prompts:

MethodHigherLowerNot different
Keyword1451
Diverse Outline1226

These results show that the quality impact varies across settings, while our methods continue to demonstrate improvements in output diversity. We will update the quality scores in the manuscript.

Comments on the paper

1 comment

Thank you for the detailed responses and additional experiments which largely address my concerns. Although the methods are simple, these findings offer practical value. I will keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.