- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper posed a framework (Diversity Injection Framework) that groups methods of improving model diversity at inference-time into three levels, based on what the output is conditioned on: nothing (level 0), a source with no semantic content (level 1, eg. a random string), and specification of the output (level 2). It also defined a transmission score for measuring how much of the variation in the source ends up in the output. It then proposed two level 2 methods (Diverse Outline and Keyword) that improve model diversity by generating diverse output specifications before getting to the final output. Both achieved higher diversity than existing methods across 5 open-ended tasks.
Critical strengths
- The proposed methods are simple and the improvement on model diversity is consistent. Across 5 open-ended tasks, both methods get better results than baselines, while the quality is comparable.
Critical weaknesses
- I think the novelty and contribution of the framework and two methods are narrower than claimed. Section 3.2 said “there is no existing test-time method provides per-output specifications to promote diversity in general open-ended generation”. Intent Factored Generation (https://arxiv.org/pdf/2506.09659) is close to what the paper described as a level 2 method.
- The transmission score analysis setup has some issues. (1) The analysis only compares with a single baseline SSOT, since the author found no existing level 2 method. However, another level 1 baseline (Random Concept) is not evaluated. (2) The comparison with SSOT itself is also not meaningful. SSOT is designed to use a random string as a source of randomness. It doesn’t need to have the string recovered from the output. So the transmission score is ~0, which doesn't show the source is ignored or level 2 is better.
Questions for authors
- The paper’s title asks “where you inject diversity matters”. Could the author elaborate on this idea? The paper reads more about “what diversity is injected”, as the main contribution (the framework) categorizes methods based on the specification of output in the source.
4 comments
Part 2 of 3: the transmission analysis (W2)
We thank the reviewer for these comments. We would like to clarify that the transmission score is intended as a diagnostic measure of the relationship between injected sources and final outputs. Specifically, it quantifies the fraction of source information that is recoverable from the output. A low transmission score does not necessarily imply that the source is ignored. Rather, it indicates that only a small fraction of the source information is recoverable from the output, although the source may still influence the generation process.
Additional Random Concept baseline. We evaluated Random Concept (Agrawal & Goyal, 2026) on Qwen3-4B and GPT-5-mini across Infinite Chats and NoveltyBench. Following its random-word injection strategy, we prepend a randomly sampled word from the wonderwords package used by the paper to each task and generate 20 responses per prompt.
The results are:
| Backbone, dataset | Method | LLM Div | Emb Div |
|---|---|---|---|
| Qwen3-4B, Infinite Chats | Direct | 2.32 | 0.19 |
| Qwen3-4B, Infinite Chats | Random Concept | 2.48 | 0.15 |
| Qwen3-4B, Infinite Chats | Diverse Outline | 6.44 | 0.35 |
| Qwen3-4B, NoveltyBench | Direct | 2.72 | 0.30 |
| Qwen3-4B, NoveltyBench | Random Concept | 2.66 | 0.22 |
| Qwen3-4B, NoveltyBench | Diverse Outline | 5.54 | 0.41 |
| GPT-5-mini, Infinite Chats | Direct | 3.40 | 0.26 |
| GPT-5-mini, Infinite Chats | Random Concept | 2.92 | 0.17 |
| GPT-5-mini, Infinite Chats | Diverse Outline | 8.01 | 0.40 |
| GPT-5-mini, NoveltyBench | Direct | 3.92 | 0.41 |
| GPT-5-mini, NoveltyBench | Random Concept | 3.31 | 0.24 |
| GPT-5-mini, NoveltyBench | Diverse Outline | 8.33 | 0.50 |
| Random Concept does not consistently improve diversity over Direct Generation in these settings, whereas Diverse Outline achieves substantially higher diversity. Embedding diversity shows a similar pattern. |
For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time.
Using the revised estimation procedure described in our response to HWs2, we obtain the following transmission scores on Qwen3-4B:
| Method | Infinite Chats | NoveltyBench |
|---|---|---|
| Diverse Outline | 0.82 | 0.89 |
| Random Concept | 0.37 | 0.47 |
| SSoT | 0.002 | 0.014 |
These results illustrate different degrees of recoverability of source information in the final outputs, with Diverse Outline achieving higher estimated transmission than Random Concept and SSoT. Importantly, the transmission score complements rather than replaces direct measures of output diversity.
Part 3 of 3: "where" versus "what" (Question)
Thank you for raising this interesting question. By “where diversity is injected,” we refer to the point in the generation process at which variation is introduced to encourage different outputs. We also recognize that “what information is injected” is another valid perspective on our framework. In fact, we view “where” and “what” as closely related: the stage at which diversity is introduced naturally influences the type of information that can be injected and how it shapes the final outputs.
Response to authors' rebuttal
Thank you for the detailed responses and additional experiments/analysis! I appreciate it and it did help strengthen the empirical comparison. However, I still think the contribution is incremental. I will keep my score.
Part 1 of 3: novelty and Intent Factored Generation (W1)
We thank the reviewer for highlighting Intent Factored Generation (IFG). We would like to clarify that our primary contribution is not the use of intermediate specifications itself, but a unified framework for characterizing diversity injection, together with a transmission-based analysis of how variation in diversity sources is reflected in final outputs. Our work systematically investigates the relationship between specification diversity and output diversity through controlled experiments, providing insights that might guide the design of future diverse generation methods. IFG can be naturally categorized as a Level 2 method under our framework. IFG samples semantic intents independently for individual responses, while Diverse Outline and Keyword diversify a set of intermediate specifications through joint generation. We additionally evaluated IFG on Qwen3-4B using NoveltyBench, with 100 prompts and 20 outputs per prompt. We followed the authors' few-shot prompting procedure, adapting the examples to our tasks, and tested two intent/response temperature configurations: (1.0, 0.5) and (1.2, 0.7).
For rows that also appear in the paper, the reported values are re-scored in the current evaluation. Note that these may differ slightly from the original paper values due to API behavior changes over time.
Keyword and Diverse Outline achieve substantially higher LLM-judged and structural diversity than IFG in this experiment, while IFG achieves higher embedding diversity at the higher-temperature configuration. These results further support the effectiveness of constructing diverse specifications at the output-set level.