- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes treating LLM generation diversity as an instruction-following problem: by repeatedly making randomized, context-dependent semantic decisions, Gacha Decoding generates diverse outputs without relying heavily on randomness in the model's token distribution. The central claim is that generation diversity can improve with model capability, rather than deteriorate as models become better aligned, and the greedy-decoding experiment demonstrate that externalized randomness can replace token-level stochasticity.
Critical strengths
- The paper offers an interesting conceptual shift from token-level stochasticity to semantic-level exploration. By decoupling idea enumeration from random selection and conditioning subsequent choices on earlier decisions, Gacha Decoding exploits LLM instruction-following capabilities rather than purely relying on token sampling. Empirical results demonstrate diversity improvements across multiple open-ended domains and model families.
Critical weaknesses
-
The exploration space remains rigid and bounded by the model's predefined semantic decomposition. Although Gacha randomizes semantic decisions, both the decision axes and their candidate options are generated by the same underlying model, potentially shifting mode collapse from final responses to the enumeration of possibilities. The decision axes are fixed before generation, with no analysis of their variability across samples. Under greedy decoding, the initial axes should be identical for a given task, although subsequent candidate lists can adapt to randomly selected decisions. Moreover, the method uses fixed numbers of axes (4–6) and requires 24–32 options per axis, regardless of task complexity or the number of meaningful alternatives. This may introduce redundant or artificial choices and unnecessarily constrain the exploration space. Importantly, decisions that emerge during reasoning cannot be incorporated into the initial schema. An adaptive mechanism that discovers decision points during generation and adjusts the branching factor according to meaningful alternatives would be more natural, particularly for complex reasoning tasks.
-
The efficiency and quality claims are not sufficiently supported by budget-matched evaluation. Each Gacha response requires 7–9 sequential LLM calls, whereas list prompting can generate up to 32 responses in one call. Consequently, the reported improvement in sample efficiency does not establish computational efficiency. Additionally, the quality-gated diversity metric masks a significant decrease in the proportion of valid generations: on InfinityChat with GPT-OSS-120B, approximately 37% of Gacha responses fail the quality threshold, compared with 8% for direct sampling (Table 2). This raises concerns about the practical quality–diversity tradeoff. The claim of uniquely favorable scaling with model capability also deserves further clarifications, as list prompting approaches Gacha's diversity on some tasks (e.g., within 2–3% on Qwen3.5-27B for chat and image generation under standard sampling). The paper does not establish whether the gains and novel modes discovery translate to objectively verifiable reasoning tasks, where randomly selected strategies may compromise correctness.
Questions for authors
-
How much of the observed diversity comes from genuinely different exploration strategies, rather than combinations of a relatively fixed set of model-preferred decisions? Could the authors quantify the diversity and coverage of the generated decision axes themselves, and investigate adaptive alternatives that dynamically identify branching points, adjust the number of options, and terminate exploration when additional decisions no longer produce meaningful variation?
-
How does Gacha Decoding compare with generating a larger pool of responses using direct or list prompting, followed by diversity-aware selection, when total inference tokens and quality-filtering costs are controlled? Would the method improve pass@k on mathematical reasoning or code generation, where diversity must translate into additional correct solutions rather than merely distinct outputs?
7 comments
Thanks for your thoughtful review! We address your token efficiency point below (2/4)
[W3, Q3] The evaluation is not token matched; Gacha Decoding may be more sample-efficient than baselines but less token-efficient.
[A3] This is a great point. To respond briefly: we optimized Gacha’s implementation and studied its token efficiency on both long-form creative writing and short-form chat (see table below). On long-form writing, we find Gacha is already more diverse from a token efficiency standpoint than list prompting and persona. On short-form chat, Gacha achieves a strictly higher diversity ceiling than list prompting (see new Figure 6), so that no amount of list samples is likely to reach the diversity of Gacha.
Table: we compare the diversity of 128 Gacha samples against list, persona, and direct sampling run to an equivalent token budget (also see new Section 4.2). Even though baselines use fewer tokens per sample, they may still reach much lower total diversity despite drawing more samples – their samples are more redundant.
Token-matched diversity comparison: long-form creative writing
| Method | Output tokens / prompt | Samples / prompt (mean) | Quality-gated diversity |
|---|---|---|---|
| Gacha Decoding | 516k | 128 | 8.49 |
| List prompting | 516k | 1,400 | 5.37 |
| Persona prompting | 517k | 253 | 7.74 |
| Direct sampling | 516k | 271 | 6.01 |
Token-matched diversity comparison: short-form InfinityChat
| Method | Output tokens / prompt | Samples / prompt (mean) | Quality-gated diversity |
|---|---|---|---|
| Gacha Decoding | 331k | 128 | 11.28 |
| List prompting | 331k | 4,335 | 16.63 |
| Persona prompting | 331k | 320 | 8.00 |
| Direct sampling | 331k | 559 | 2.71 |
| Higher budgets (list saturates): | |||
| Gacha Decoding | 2.39M | 768 | 18.55 |
| List prompting | 6.91M | 65,536 | 17.79 |
In more detail: we agree that token efficiency is distinct from sample efficiency. Our initial submission prioritized per-sample diversity, as we believe it is the right metric in many use cases (for example, if many users independently ask the model a similar query, we may want each user to get different single responses to avoid homogenization). Nonetheless, we agree token efficiency is also a key consideration. Our existing runs did not log reasoning tokens, so we reran our main experiments on Inkling (the best open-weight model considered), counting total tokens (including thinking, intermediate planning, etc.) for each response. Spurred by your feedback, we also optimized our Gacha Decoding implementation by (1) caching/sharing the decision schema and first option menu across batches of 16 samples to amortize planning and (2) lowering thinking effort during planning. This makes Gacha 3.6x cheaper, with largely unchanged diversity/quality (going from 7.0x → ~2x the token cost of persona or direct sampling). See the table above for our token-matched diversity comparisons.
On long-form creative writing, Gacha is the most token-efficient method, achieving the highest diversity at matched tokens. On short-form chat, Gacha beats persona prompting but trails list on token efficiency at N=128, since list can produce 32 responses per call while Gacha and persona require at least one call per response. This advantage is specific to short-form tasks: on creative writing, requesting more than 8 responses per list call yields invalid story snippets.
Even on short-form tasks, Gacha soon overtakes list, as Gacha exhibits a significantly higher diversity ceiling (Fig 6): list’s diversity saturates, such that even 65536 list samples cannot match just 768 Gacha samples, and no practical amount of further list scaling is likely to close this gap.
We have updated our paper to include these analyses (Section 4.2, optimization details in Appendix C.4). Thank you for helping us improve our work!
Thanks for your thoughtful review! We address your points below (3/4)
[W4] The claim of Gacha Decoding’s uniquely favorable scaling with model capability should be clarified, since list prompting also scales well on some tasks.
[A4] To clarify, our claim is that only Gacha Decoding consistently scales with instruction-following capability across all domains (Figure 4 caption). We agree that list prompting also sometimes scales (L369-372), but only on Qwen with short-form domains (for example, see Figure 1 for other models where list does not scale on short-form chat). We view list prompting as a partial realization of our instruction-following-centric view of diversity: it can use instruction following to diversify within a list, which can be effective on short-form domains where many responses fit in a single list, but breaks down on longer-form domains such as creative writing (Figure 4).
[W5] There is a practical quality–diversity tradeoff, and in particular on chat with GPT-OSS-120B, the proportion of valid responses decreases significantly with Gacha Decoding.
[A5] We believe Gacha’s chat validity rate is low on GPT-OSS as it was the weakest model we considered, and Gacha’s validity increases significantly with stronger models. On GPT 5.5 for example, Gacha’s chat validity is comparable to that of baselines (89.6%, versus 91.3% for list prompting). Moreover, as shown in Table 3 (Table 2 of original submission), Gacha Decoding consistently achieves the highest average quality on creative writing, beating all baselines on GPT-OSS, Opus 4.8, and GPT-5.5. While our diversity metric already discounts invalid responses so that any quality drop counts against a method in our numbers, we agree that an equal quality threshold is not the same as equal overall quality. Based on your feedback, we have revised our abstract to make clear that our results refer to the former – thanks for helping us clarify our work!
[W6, Q4] The paper does not establish gains on verifiable tasks, for example pass@k on code and math.
[A6] We focus our evaluation on open-ended creative tasks. Diversity for math and code reasoning is out of scope for this work, but we are optimistic that future work can show benefits of Gacha Decoding on reasoning-intensive tasks. As an initial result in this direction, we showed that Gacha can improve the diversity of successful de novo protein designs, where success is judged by predicted binding confidence, following prior work (L407). We have updated Section 6 to discuss this promising future direction.
Thanks for your thoughtful review! We address your points below (1/4)
Thanks for your thoughtful review! We are glad you found our work an “interesting” conceptual shift away from token-stochasticity based diversity. We address your points below, and have revised our paper to incorporate your feedback.
[W1] Gacha randomizes semantic decisions, but the options for each decision are generated as a list from the same underlying model. This potentially shifts mode collapse from final responses to option enumeration.
[A1] Thanks for your point! To clarify a potential misunderstanding, Gacha Decoding does not rely on diversity across independent LM enumeration calls, so mode collapse in option enumeration does not hinder our method. As our greedy decoding study shows, even with completely collapsed sampling, Gacha Decoding still maintains strong output diversity, and in fact Gacha run with greedy decoding often outperforms baselines run with sampling.
Gacha does rely on diversity within each option enumeration call, i.e., we prompt the LM to generate k options and expect k distinct outputs within the same list. We observe that this is an instruction following task that the LM can perform even if the LM is fully collapsed.
Based on your suggestion, we also directly evaluated the diversity of the generated decision options, both within and across option lists. We find that the model is able to generate distinct options within each list (i.e. it can follow instructions), and that sequential conditioning significantly increases diversity across lists at later points in the trajectory. This matches your intuition that subsequent candidate lists adapt to randomly selected decisions, and we believe it is a large part of how Gacha Decoding maintains enumeration diversity under greedy decoding. Please see our answer [A7] to your Q1 for this analysis.
[W2, Q2] Gacha Decoding fixes the decisions for planning the response upfront. This may unnecessarily constrain exploration, compared to adaptive planning and discovering decision points during generation.
[A2] Our existing recipe already strongly outperforms baselines, but we agree that there are exciting ways to build on Gacha and improve it further. Adaptive planning is one promising direction. Spurred by your suggestion, we implemented and tested an adaptive extension of Gacha. Instead of fixing the decision schema up front, at each step the model reasons about which decision to make next and generates an option list of whatever size it judges appropriate, after which one option is selected at random. We also tried a further extension in which the model proposes several candidate decisions at each step, and the decision itself is selected at random too. As the table below shows (results on Inkling, N=128 samples per prompt), our original recipe outperforms both adaptive variants. A key factor is that when the model chooses each decision dynamically, its decisions tend to become overly specific, which narrows the space it explores. We view getting adaptive planning to work well as exciting future work, and have added this note to Section 6.