acceptodds
Under review as a conference paper at ICLR 2027

Gacha Decoding: Eliciting Diverse Generations Through Instruction Following

Abstract

Language models often produce surprisingly homogeneous responses to open-ended prompts, with a model's diversity tending to decline as its capability increases. We introduce Gacha Decoding, an inference-time method for eliciting diverse language model generations. Across open-ended domains (in-the-wild chat, creative writing, planning for image generation, and protein design), Gacha Decoding significantly outperforms existing approaches in diversity at equal quality (up to 2.3x Vendi over the next-best prior approach), reaching the same number of high-quality modes with over an order of magnitude fewer samples (11.0x), and discovering novel modes that no other approach surfaces. Our key insight is to treat diversity as an instruction-following problem: rather than relying on the LM's token entropy, we combine its instruction-following capability with randomness from an external RNG tool to scalably identify and realize distinct modes of the response space. This approach of "planning with dice" enables Gacha to invert the long-observed tension between diversity and model capability. As the underlying LM becomes a better instruction follower, diversity under Gacha Decoding consistently improves–even as its entropy decreases and diversity under prior approaches declines. Together, our results highlight that instruction following, rather than token entropy alone, can serve as the engine driving response diversity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
generation diversity, mode collapse, diverse sampling, inference-time decoding

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_u8QzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes treating LLM generation diversity as an instruction-following problem: by repeatedly making randomized, context-dependent semantic decisions, Gacha Decoding generates diverse outputs without relying heavily on randomness in the model's token distribution. The central claim is that generation diversity can improve with model capability, rather than deteriorate as models become better aligned, and the greedy-decoding experiment demonstrate that externalized randomness can replace token-level stochasticity.

Critical strengths
  • The paper offers an interesting conceptual shift from token-level stochasticity to semantic-level exploration. By decoupling idea enumeration from random selection and conditioning subsequent choices on earlier decisions, Gacha Decoding exploits LLM instruction-following capabilities rather than purely relying on token sampling. Empirical results demonstrate diversity improvements across multiple open-ended domains and model families.
Critical weaknesses
  • The exploration space remains rigid and bounded by the model's predefined semantic decomposition. Although Gacha randomizes semantic decisions, both the decision axes and their candidate options are generated by the same underlying model, potentially shifting mode collapse from final responses to the enumeration of possibilities. The decision axes are fixed before generation, with no analysis of their variability across samples. Under greedy decoding, the initial axes should be identical for a given task, although subsequent candidate lists can adapt to randomly selected decisions. Moreover, the method uses fixed numbers of axes (4–6) and requires 24–32 options per axis, regardless of task complexity or the number of meaningful alternatives. This may introduce redundant or artificial choices and unnecessarily constrain the exploration space. Importantly, decisions that emerge during reasoning cannot be incorporated into the initial schema. An adaptive mechanism that discovers decision points during generation and adjusts the branching factor according to meaningful alternatives would be more natural, particularly for complex reasoning tasks.

  • The efficiency and quality claims are not sufficiently supported by budget-matched evaluation. Each Gacha response requires 7–9 sequential LLM calls, whereas list prompting can generate up to 32 responses in one call. Consequently, the reported improvement in sample efficiency does not establish computational efficiency. Additionally, the quality-gated diversity metric masks a significant decrease in the proportion of valid generations: on InfinityChat with GPT-OSS-120B, approximately 37% of Gacha responses fail the quality threshold, compared with 8% for direct sampling (Table 2). This raises concerns about the practical quality–diversity tradeoff. The claim of uniquely favorable scaling with model capability also deserves further clarifications, as list prompting approaches Gacha's diversity on some tasks (e.g., within 2–3% on Qwen3.5-27B for chat and image generation under standard sampling). The paper does not establish whether the gains and novel modes discovery translate to objectively verifiable reasoning tasks, where randomly selected strategies may compromise correctness.

Questions for authors
  1. How much of the observed diversity comes from genuinely different exploration strategies, rather than combinations of a relatively fixed set of model-preferred decisions? Could the authors quantify the diversity and coverage of the generated decision axes themselves, and investigate adaptive alternatives that dynamically identify branching points, adjust the number of options, and terminate exploration when additional decisions no longer produce meaningful variation?

  2. How does Gacha Decoding compare with generating a larger pool of responses using direct or list prompting, followed by diversity-aware selection, when total inference tokens and quality-filtering costs are controlled? Would the method improve pass@k on mathematical reasoning or code generation, where diversity must translate into additional correct solutions rather than merely distinct outputs?

7 comments
AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your points below (1/4)

Thanks for your thoughtful review! We are glad you found our work an “interesting” conceptual shift away from token-stochasticity based diversity. We address your points below, and have revised our paper to incorporate your feedback.

[W1] Gacha randomizes semantic decisions, but the options for each decision are generated as a list from the same underlying model. This potentially shifts mode collapse from final responses to option enumeration.

[A1] Thanks for your point! To clarify a potential misunderstanding, Gacha Decoding does not rely on diversity across independent LM enumeration calls, so mode collapse in option enumeration does not hinder our method. As our greedy decoding study shows, even with completely collapsed sampling, Gacha Decoding still maintains strong output diversity, and in fact Gacha run with greedy decoding often outperforms baselines run with sampling.

Gacha does rely on diversity within each option enumeration call, i.e., we prompt the LM to generate kk options and expect kk distinct outputs within the same list. We observe that this is an instruction following task that the LM can perform even if the LM is fully collapsed.

Based on your suggestion, we also directly evaluated the diversity of the generated decision options, both within and across option lists. We find that the model is able to generate distinct options within each list (i.e. it can follow instructions), and that sequential conditioning significantly increases diversity across lists at later points in the trajectory. This matches your intuition that subsequent candidate lists adapt to randomly selected decisions, and we believe it is a large part of how Gacha Decoding maintains enumeration diversity under greedy decoding. Please see our answer [A7] to your Q1 for this analysis.

[W2, Q2] Gacha Decoding fixes the decisions for planning the response upfront. This may unnecessarily constrain exploration, compared to adaptive planning and discovering decision points during generation.

[A2] Our existing recipe already strongly outperforms baselines, but we agree that there are exciting ways to build on Gacha and improve it further. Adaptive planning is one promising direction. Spurred by your suggestion, we implemented and tested an adaptive extension of Gacha. Instead of fixing the decision schema up front, at each step the model reasons about which decision to make next and generates an option list of whatever size it judges appropriate, after which one option is selected at random. We also tried a further extension in which the model proposes several candidate decisions at each step, and the decision itself is selected at random too. As the table below shows (results on Inkling, N=128 samples per prompt), our original recipe outperforms both adaptive variants. A key factor is that when the model chooses each decision dynamically, its decisions tend to become overly specific, which narrows the space it explores. We view getting adaptive planning to work well as exciting future work, and have added this note to Section 6.

InfinityChatCreative Writing
Gacha Decoding11.698.44
Adaptive Gacha10.108.20
Adaptive Gacha + Randomize over what decision to make10.578.12
AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your token efficiency point below (2/4)

[W3, Q3] The evaluation is not token matched; Gacha Decoding may be more sample-efficient than baselines but less token-efficient.

[A3] This is a great point. To respond briefly: we optimized Gacha’s implementation and studied its token efficiency on both long-form creative writing and short-form chat (see table below). On long-form writing, we find Gacha is already more diverse from a token efficiency standpoint than list prompting and persona. On short-form chat, Gacha achieves a strictly higher diversity ceiling than list prompting (see new Figure 6), so that no amount of list samples is likely to reach the diversity of Gacha.

Table: we compare the diversity of 128 Gacha samples against list, persona, and direct sampling run to an equivalent token budget (also see new Section 4.2). Even though baselines use fewer tokens per sample, they may still reach much lower total diversity despite drawing more samples – their samples are more redundant.

Token-matched diversity comparison: long-form creative writing

MethodOutput tokens / promptSamples / prompt (mean)Quality-gated diversity
Gacha Decoding516k1288.49
List prompting516k1,4005.37
Persona prompting517k2537.74
Direct sampling516k2716.01

Token-matched diversity comparison: short-form InfinityChat

MethodOutput tokens / promptSamples / prompt (mean)Quality-gated diversity
Gacha Decoding331k12811.28
List prompting331k4,33516.63
Persona prompting331k3208.00
Direct sampling331k5592.71
Higher budgets (list saturates):
Gacha Decoding2.39M76818.55
List prompting6.91M65,53617.79

In more detail: we agree that token efficiency is distinct from sample efficiency. Our initial submission prioritized per-sample diversity, as we believe it is the right metric in many use cases (for example, if many users independently ask the model a similar query, we may want each user to get different single responses to avoid homogenization). Nonetheless, we agree token efficiency is also a key consideration. Our existing runs did not log reasoning tokens, so we reran our main experiments on Inkling (the best open-weight model considered), counting total tokens (including thinking, intermediate planning, etc.) for each response. Spurred by your feedback, we also optimized our Gacha Decoding implementation by (1) caching/sharing the decision schema and first option menu across batches of 16 samples to amortize planning and (2) lowering thinking effort during planning. This makes Gacha 3.6x cheaper, with largely unchanged diversity/quality (going from 7.0x → ~2x the token cost of persona or direct sampling). See the table above for our token-matched diversity comparisons.

On long-form creative writing, Gacha is the most token-efficient method, achieving the highest diversity at matched tokens. On short-form chat, Gacha beats persona prompting but trails list on token efficiency at N=128, since list can produce 32 responses per call while Gacha and persona require at least one call per response. This advantage is specific to short-form tasks: on creative writing, requesting more than 8 responses per list call yields invalid story snippets.

Even on short-form tasks, Gacha soon overtakes list, as Gacha exhibits a significantly higher diversity ceiling (Fig 6): list’s diversity saturates, such that even 65536 list samples cannot match just 768 Gacha samples, and no practical amount of further list scaling is likely to close this gap.

We have updated our paper to include these analyses (Section 4.2, optimization details in Appendix C.4). Thank you for helping us improve our work!

AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your points below (3/4)

[W4] The claim of Gacha Decoding’s uniquely favorable scaling with model capability should be clarified, since list prompting also scales well on some tasks.

[A4] To clarify, our claim is that only Gacha Decoding consistently scales with instruction-following capability across all domains (Figure 4 caption). We agree that list prompting also sometimes scales (L369-372), but only on Qwen with short-form domains (for example, see Figure 1 for other models where list does not scale on short-form chat). We view list prompting as a partial realization of our instruction-following-centric view of diversity: it can use instruction following to diversify within a list, which can be effective on short-form domains where many responses fit in a single list, but breaks down on longer-form domains such as creative writing (Figure 4).

[W5] There is a practical quality–diversity tradeoff, and in particular on chat with GPT-OSS-120B, the proportion of valid responses decreases significantly with Gacha Decoding.

[A5] We believe Gacha’s chat validity rate is low on GPT-OSS as it was the weakest model we considered, and Gacha’s validity increases significantly with stronger models. On GPT 5.5 for example, Gacha’s chat validity is comparable to that of baselines (89.6%, versus 91.3% for list prompting). Moreover, as shown in Table 3 (Table 2 of original submission), Gacha Decoding consistently achieves the highest average quality on creative writing, beating all baselines on GPT-OSS, Opus 4.8, and GPT-5.5. While our diversity metric already discounts invalid responses so that any quality drop counts against a method in our numbers, we agree that an equal quality threshold is not the same as equal overall quality. Based on your feedback, we have revised our abstract to make clear that our results refer to the former – thanks for helping us clarify our work!

[W6, Q4] The paper does not establish gains on verifiable tasks, for example pass@k on code and math.

[A6] We focus our evaluation on open-ended creative tasks. Diversity for math and code reasoning is out of scope for this work, but we are optimistic that future work can show benefits of Gacha Decoding on reasoning-intensive tasks. As an initial result in this direction, we showed that Gacha can improve the diversity of successful de novo protein designs, where success is judged by predicted binding confidence, following prior work (L407). We have updated Section 6 to discuss this promising future direction.

Reviewer_kYVeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposed a new method (Gacha Decoding) for LLMs to generate more diverse answers to open-ended tasks at inference time. The main conceptual contribution is, instead of improving diversity from the perspective of token entropy, it sees diversity as an instruction-following problem. For each task, the model first plans a trajectory of decision steps. At each step, it enumerates a list of options, conditioned on previous steps’ options. Instead of asking the model itself to choose from the option, Gacha uses an external random number generator (RNG) to pick one. The model then writes the final response based on all the selected choices at each step. On three evaluations (in the wild chat, creative writing, image generation), it achieved higher diversity than previous methods, with matched quality. The response diversity also increases with the model's instruction-following ability, given IFBench scores across Qwen 3.5 sizes. An ablation study shows that each of the design in Gacha (working on ideas instead of tokens, external RNG, conditioned on earlier choices) is necessary.

Critical strengths
  1. The method shows consistent strong performance against existing methods. Even with quality-gated evaluation, the method achieves higher diversity than baselines across multiple domains. The paper’s ablation study also shows the necessity of every design in the pipeline, showing a robust method.
  2. The concept of viewing diversity in the perspective of ideas with an external RNG instead of token entropy is interesting. It shows little loss on diversity even under greedy decoding. This can be applied as a harness to API models with no need for training.
Critical weaknesses
  1. The method turns response to an open-ended task into a trajectory of multiple decision steps, and each step the model needs to enumerate a list of plausible options. So the method depends on model’s ability to create a diverse list in the first place, which the paper itself noted in Section 6. However this is not directly evaluated. Addressing this will strengthen the paper.
Questions for authors
  1. If the trajectory has t steps and each step requires a list of k options, the total number of possible combinations is k^t. I wonder how the final diversity changes with t and k?
5 comments
AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your points below

Thanks for your positive and thoughtful review! We are glad you liked our conceptual contribution of reframing diversity away from token entropy. We address your points below, and have revised our paper to incorporate your feedback.

[W1] As Section 6 notes, Gacha Decoding generates responses via a trajectory of decision steps, where each step assumes that the model can generate a diverse list of decision options. Evaluating this assumption would strengthen the paper.

[A1] Great suggestion! The strength of our list-prompting baseline (which surprisingly, often beats all existing approaches in the literature) strongly suggests that the models we study can generate diverse lists. Spurred by your feedback, we further analyzed the assumption that models can generate diverse decision options for Gacha Decoding directly using our existing Gacha traces from Opus 4.8 on chat (100 prompts × subset of 16 trajectories per prompt). Here, Gacha makes 4 decisions per prompt and calls the LM to list at least 24 options per decision (more is possible). We embed the generated options at each decision step with Qwen3-4B-Embedding and test:

Within-list diversity: do the options within a single decision differ from one another? Yes. We measure embedding distances between options within a list, and find that on average, each list yields 25.0 distinct options out of 26.1 generated. We define a duplicate pair as one whose cosine similarity is above 0.9, a threshold we set conservatively based on manual inspection (pairs such as “Early 1990s” and “Late 1990s” are counted as duplicates). The overall mean pairwise cosine distance within a list is 0.38, compared to a rough ceiling of 0.56 for options drawn at random across unrelated prompts.

Across-list diversity: do different trajectories to the same prompt enumerate different options at the same decision step? Yes, increasingly so as the trajectory progresses. For each prompt, we take 16 independently sampled Gacha trajectories. At each decision step, every trajectory produces its own option list, conditioned on the prompt and earlier choices within the trajectory. We report (1) the share of options that have a duplicate (cosine similarity > 0.9) anywhere in the other 15 trajectories' lists at that step, and (2) the mean cosine distance (higher = more diverse) between pairs of options drawn from different trajectories' lists at that step.

Decision step1234
Options with a duplicate in another trajectory's list64.1%29.0%20.8%14.6%
Mean pairwise cosine distance across lists0.4310.4500.4640.474

At the first decision, list generation is conditioned only on the decision schema and the shared task prompt, so variation in the first list across calls is largely dependent on token entropy. Unsurprisingly, the model tends to re-enumerate similar lists: 64% of options reappear in another trajectory. Repetition falls sharply once trajectories diverge through random decisions, to 15% by the fourth decision. Diversity across planning trajectories thus benefits heavily from conditioning later lists on earlier random choices. This reinforces our ablation finding that sequential conditioning is important. We have added this analysis to Appendix A.5, thank you for helping us strengthen our paper!

[Q1] Can the diversity from Gacha Decoding scale with the number of steps tt in the planning trajectory, or the number of options kk considered at each step?

[A2] Great question! We repeated our main experiments on chat across varying choices of kk and tt with Inkling (the strongest open-weight model considered in our work) at n=256n=256 generations per prompt. Overall, we observe positive scaling in both tt and kk (see results below, reporting quality-gated Vendi with validity rate in parentheses):

Varying the number of steps (k=24k = 24 options per step, as in our original setup):

Steps tt1234 (original)56
Diversity11.28 (89%)12.94 (83%)13.82 (79%)14.38 (74%)14.48 (70%)14.24 (66%)

Varying the number of options per step (t=4t = 4 steps, as in our original setup):

Options kk481624 (original)3240
Diversity12.24 (82%)13.24 (79%)14.08 (76%)14.38 (74%)14.50 (72%)14.51 (70%)

Diversity grows quickly with tt and kk when ktk^t is small, and levels off at around 14.5 once ktk^t reaches roughly 10510^5 to 10610^6. Much larger values of ktk^t do not increase diversity further: generating with 6 decision steps and 32 options per step (kt≈109k^t \approx 10^9) yields 14.35 quality-gated Vendi. We view further scaling to longer trajectories (e.g., by training with the Gacha harness) as exciting future work.

We have included these results in an updated version of our paper (Appendix A.5). Thanks for helping strengthen our analysis!

AuthorsOpenReview ↗

Thanks for your thoughtful review! We look forward to further discussion with you

Thank you again for your review. Please let us know if you have any further questions about our work. We look forward to discussing with you!

Reviewer_TGnhOpenReview ↗

Response to the authors' rebuttal

Thank you to the authors for the thorough response to my concerns and for running the extra experiments as part of their revision.

I’m increasing my rating from 3 to 4. The token-matched comparison was the main thing that changed my mind. Gacha Decoding still leads on long-form writing after counting the planning and reasoning tokens. For short chat, I actually found it useful to see that list prompting wins at a lower token budget, but Gacha Decoding reaches higher diversity at larger budgets. To a reader/practiitoner, this tells when to use each method, which was not possible from the original sample-count comparison.

I would still think the method is an incremental step over prior work, but the new ablation results do indeed make a better case that sequential conditioning contributes to the gains. The quality threshold sweep and the more precise abstract also address my main concerns.

Reviewer_TGnhOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Gacha Decoding is a method for for getting diverse outputs from an LLM. The model first identifies semantic decisions relevant to the task, lists possible choices, and uses external randomness to pick one. Later choices are generated conditioned on the earlier selections. Finally, the model produces a response following these choices. This work evaluates this on different task settings - open-ended chat, creative writing, image generation via prompt expansion, and a protein-design pilot. They report improvements in quality-filtered Vendi diversity, and ablations and experiments showing that the method retains diversity under greedy decoding.

Critical strengths

1 - The empirical results across models and domains are interesting. The method improves quality-filtered diversity even over a tuned list-prompting baseline, which turns out to be quite strong. I also liked the image-generation experiments and protein-design pilot. Protein-design is based on predicted structures and interface confidence, but still an interesting application beyond text generation.

2 - I like the connection to the tension between model capabilities, instruction following, and output diversity. In particular, the greedy-decoding results make the point quite clearly: a model can still produce diverse outputs when randomness comes from the surrounding harness. The ablations on external randomness and enumeration help support this.

Critical weaknesses

1 - I find the conceptual novelty somewhat limited. Generating diverse plans/specifications and then conditioning outputs on them has been explored in prior work - Luminate, Intent Factored Generation, and Where You Inject Diversity Matters. The more specific contribution here seems to be sequentially generating options conditioned on previous choices, combined with external random selection. This is a useful recipe, but the broader framing feels a bit stronger than the novelty warrants. The independent-axes ablation helps, though the advantage from sequential conditioning is small in some settings.

2 - Some claims in the abstract need clearer qualification. “11× fewer samples” counts final outputs, while each Gacha output requires several intermediate LLM calls. This leaves the actual cost advantage unclear. Against the stronger list baseline, the highlighted chat result is also 4.7× rather than 11×. Similarly, applying a common quality threshold does not establish equal overall quality, especially when pass rates differ. “Discovering novel modes that no other approach surfaces” also seems too broad for examples where those modes were simply absent from a finite set of baseline samples.

Questions for authors

1 - What is the total token usage/cost for reaching comparable diversity with Gacha, list prompting, and persona prompting, including the intermediate calls? Can this be reported from the existing runs?

2 - How close is the independent-axes ablation to the Keyword method in Where You Inject Diversity Matters? Also, how much uncertainty is there around the smaller gains from sequential conditioning?

3 - How sensitive are the results to the quality threshold? Given the pass-rate differences in Table 2, can you clarify what exactly is meant by “equal quality”? A comparison at a few thresholds using the existing outputs would help.

5 comments
AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your novelty point below (1/3)

Thanks for your positive and thoughtful review! We are glad you liked our greedy decoding experiment and found our empirical results interesting. We address your points below, and have revised our paper to incorporate your feedback.

[W1] What is the key conceptual novelty? Generating diverse plans and conditioning responses on them has also been explored in prior work.

[A1] Thanks for your point. We agree that conditioning on diverse plans is a common abstraction for improving diversity that we build on (L190-193, L520-522). We see our core novelty as follows:

Conceptually, we are the first to our knowledge to explicitly connect and analyze the relationship between diversity and IF capabilities. We show how this connection can be leveraged to reverse a widely-reported negative trend between model capability and response diversity (Figure 1, 4), a surprising result. Our controlled experiments demonstrating this connection—for example the greedy decoding study you highlighted and our diversity-versus-IF scaling study (Figure 4)—are core parts of our scientific contribution, beyond our performant recipe.

Prior plan-based diverse generation works do not take our IF-centric view. Intent-Factored Generation [1] diversifies plans via temperature sampling (L1239-1241). Where You Inject Diversity Matters [2] proposes a Diverse Outline method, which generates a list of ideas and relies on token entropy for diversity across lists when scaled (L956-957). The closest prior method to Gacha Decoding is the Keyword method also from [2], which is analogous to our independent-axes ablation. However, they did not identify or study the connection between instruction following and diversity. Our full Gacha recipe introduces sequential conditioning, which further leverages our conceptual shift towards instruction following. This enables Gacha Decoding to (a) improve significantly over the Keyword-analogous independent-axes ablation on creative writing and chat, (b) match or significantly improve on image generation, and (c) reach a strictly higher diversity ceiling (see also our answer [A4] to Q2 below).

We have revised our related work and ablation sections to make our positioning more clear, and have also updated to cite Luminate [3], which shows via user studies that a method similar to Keyword [2] improves human-AI co-writing. Thank you for pointing out this relevant work!

[1] Intent Factored Generation, Ahmed et al., NeurIPS 2026; [2] Where You Inject Diversity Matters, Zhang et al., arXiv 2026; [3] Luminate, Suh et al., CHI 2024

AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your token efficiency point below (2/3)

[W2, Q1] Generating a sample with Gacha requires a multi-turn agentic rollout, while other methods (e.g. persona, direct sampling) are single-turn. Thus, sample efficiency is not a complete cost comparison.

[A2] This is a great point. To respond briefly: we optimized Gacha’s implementation and studied its token efficiency on both long-form creative writing and short-form chat (see table below). On long-form writing, we find Gacha is already more diverse from a token efficiency standpoint than list prompting and persona. On short-form chat, Gacha achieves a strictly higher diversity ceiling than list prompting (see new Fig 6), so that no amount of list samples is likely to reach the diversity of Gacha.

Table: we compare the diversity of 128 Gacha samples against list, persona, and direct sampling run to an equivalent token budget (also see new Section 4.2). Even though baselines use fewer tokens per sample, they may still reach much lower total diversity despite drawing more samples – their samples are more redundant.

Token-matched diversity comparison: long-form creative writing

MethodOutput tokens / promptSamples / prompt (mean)Quality-gated diversity
Gacha Decoding516k1288.49
List prompting516k1,4005.37
Persona prompting517k2537.74
Direct sampling516k2716.01

Token-matched diversity comparison: short-form InfinityChat

MethodOutput tokens / promptSamples / prompt (mean)Quality-gated diversity
Gacha Decoding331k12811.28
List prompting331k4,33516.63
Persona prompting331k3208.00
Direct sampling331k5592.71
Higher budgets (list saturates):
Gacha Decoding2.39M76818.55
List prompting6.91M65,53617.79

In more detail: we agree that token efficiency is distinct from sample efficiency. Our initial submission prioritized per-sample diversity, as we believe it is the right metric in many use cases (for example, if many users independently ask the model a similar query, we may want each user to get different single responses to avoid homogenization). Nonetheless, we agree token efficiency is also a key consideration. Our existing runs did not log reasoning tokens, so we reran our main experiments on Inkling (the best open-weight model considered), counting total tokens (including thinking, intermediate planning, etc.) for each response. Spurred by your feedback, we also optimized our Gacha Decoding implementation by (1) caching/sharing the decision schema and first option menu across batches of 16 samples to amortize planning and (2) lowering thinking effort during planning. This makes Gacha 3.6x cheaper, with largely unchanged diversity/quality (going from 7.0x → ~2x the token cost of persona or direct sampling). See the table above for our token-matched diversity comparisons.

On long-form creative writing, Gacha is the most token-efficient method, achieving the highest diversity at matched tokens. On short-form chat, Gacha beats persona prompting but trails list on token efficiency at N=128, since list can produce 32 responses per call while Gacha and persona require at least one call per response. This advantage is specific to short-form tasks: on creative writing, requesting more than 8 responses per list call yields invalid story snippets. Even on short-form tasks, Gacha soon overtakes list, as Gacha exhibits a significantly higher diversity ceiling (Fig 6): list’s diversity saturates, such that even 65536 list samples cannot match just 768 Gacha samples, and no practical amount of further list scaling is likely to close this gap.

We have updated our paper to include these analyses (Section 4.2, optimization details in Appendix C.4). Thank you for helping us improve our work!

AuthorsOpenReview ↗

Thanks for your thoughtful review! We address your remaining points below (3/3)

[W3] The abstract can be revised to make the central claims clearer. For example, what method does the “11x fewer samples” refer to (the gain over a list baseline is 4.7x)? Quality thresholding also does not mean equal overall quality.

[A3] Thank you for helping us clarify our work! We’ve made the following abstract changes based on your suggestions, and are happy to discuss further edits.

  • 11× fewer samples: This refers to persona prompting, the best approach from prior work (i.e. the “next-best prior approach” in the abstract). The strength of our list baseline is a surprising finding in its own right (it contrasts with recent work) and we obtained it by carefully tuning the prompt and hyperparameters. We therefore view it as an auxiliary contribution of our paper, not as a prior approach. We have revised our abstract to name persona prompting explicitly.

  • Novel modes: We updated to say these modes were not surfaced by any baseline within our sampling budget. That said, our new scaling analysis (see A2 above) suggests this result is not an artifact of finite sampling: list’s diversity plateaus at around 8192 samples, and even 65536 list samples fall short of 768 Gacha samples. These results suggest Gacha reaches modes that baselines do not reach at practical budgets.

  • Quality thresholding: Agreed! We now describe the comparison as diversity at “equal quality threshold.”

[Q2] How does the independent-axis ablation of Gacha Decoding compare to the Keyword method in Where You Inject Diversity Matters, and can the statistical error in the observed gain be quantified?

[A4] Please see our response [A1] above, which answers your question. Our full Gacha Decoding recipe adds sequential conditioning, which we argue is a more scalable way to operationalize our core concept of diversity from instruction following, as it allows earlier random choices to influence later LM enumeration calls (L224-229). Supporting this intuition empirically, we also find that Gacha Decoding can achieve a much higher diversity ceiling than our independent-axes ablation (Fig 6).

We further ran paired t-tests over evaluation prompts on the difference between Gacha and each ablation in Table 1, which is now updated. On Opus 4.8 (Table 1), Gacha’s gains over ablations are all significant at p<0.01 (mean ± 95 CI: Gacha gains +2.41 ± 0.50 over independent-axes on chat and +1.68 ± 0.14 on creative writing), except Gacha and independent-axes on image generation where there was no significant difference. This depends on the model tested, for example, on image generation with Inkling, Gacha significantly improves over independent-axes (+1.14 ± 0.36).

[Q3] Could the authors clarify what is meant by equal quality? How would the main results change with different quality thresholds?

[A5] Good question. By “equal quality,” we mean that diversity is computed only over samples that pass a common quality threshold. We set this threshold based on the quality of the model’s direct samples (L243), so that passing responses match the quality of what the model would natively generate. This lets us measure how much diversity each method gains while holding quality near the level the model already delivers. We have revised our abstract to make this clear.

Our main results are not sensitive to this choice of quality threshold. Below we report quality-gated diversity (n=256 samples) at three absolute quality thresholds for our original Opus 4.8 outputs on creative writing and image generation (these are the two domains with continuous quality metrics). We have added this analysis to our revised paper (Appendix A.5).

Creative writing (quality score 0-20, parenthesized numbers are threshold pass rate)

MethodCurrent quality threshold (avg. 14.5)≥ 15≥ 16≥ 17
Gacha Decoding11.84 (98%)11.76 (94%)11.47 (87%)10.63 (70%)
List5.65 (19%)5.23 (16%)3.20 (5%)1.00 (1%)
Persona8.37 (84%)8.25 (78%)7.65 (59%)6.10 (29%)
Direct Sampling5.53 (79%)5.24 (66%)4.60 (42%)3.20 (16%)

Image generation (alignment score 0–1)

MethodCurrent quality threshold (avg. 0.836)≥ 0.90≥ 0.95≥ 1.0 (passes every check)
Gacha Decoding10.03 (58%)8.57 (51%)7.54 (39%)7.40 (38%)
List6.78 (66%)5.76 (59%)5.37 (47%)5.28 (45%)
Persona4.63 (75%)4.19 (66%)3.89 (53%)3.86 (52%)
Direct Sampling2.98 (76%)2.71 (64%)2.49 (52%)2.46 (50%)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.