- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces a unified mathematical framework and library for "composable" decoding. The authors formulate decoding as an optimization problem with a flexible, general regularization term that allows one to bias the output distribution in different ways. They solve this optimization problem on the simplex with mirror ascent.
They show that several well-known existing decoding methods (such as temperature, top-p, top-k, etc.) can be seen as special cases of this regularization term. However, the flexibility of the formulation allows one to further compose multiple objectives together with weights. To explore this, the authors then introduce a new decoding method, "Best-of- Decoding", which jointly optimizes for consistency and coverage.
The paper evaluates the proposed framework and decoding method in 4 diverse model-benchmark settings, and they show modest-to-large gains across GPQA, Math-500, IFEval, and LiveCodeBench, compared to vanilla single-objective decoding methods such as top-p, top-k, etc. Most notably, jointly optimizing Jenson-Shannon divergence and diversity allows them to achieve a +10.6 pp gain on GPQA pass@1 for Qwen3-4B-Base.
Critical strengths
-
The core framework is simple, clean, and quite general. This makes the space of decoding methods much easier to reason about, and the formulation is intuitive once stated.
-
The compositional interface is flexible and naturally extensible. New differentiable preferences can be introduced as regularizers, combined with existing ones, weighted, and optimized through the same general machinery, rather than requiring a new decoding algorithm for each setting. Although the particular regularizers explored in this paper are fairly simple, the framework seems broad enough to support substantially richer objectives in future work. For example, one could imagine objectives for open-ended writing or other settings with several competing desiderata that need to be balanced jointly. More broadly, I see value in having a common framework in which new decoding objectives can be incorporated and composed without changing the overall decoding machinery.
-
The empirical results show that the framework can find meaningfully better operating points. The strongest pass@1 improvements are substantial (e.g., +10.6 points on GPQA with Top-k support and +5.4 points on MATH500 with Top-k), and other settings show intuitive trade-offs between single-sample quality, multi-sample success, and diversity.
Critical weaknesses
- The empirical scope is a little limited (without trying to be "Reviewer 2"). The paper evaluates four model/benchmark pairs covering math, scientific reasoning, instruction following, and code, but does not test open-ended generation or writing quality. I would have liked to see at least one setting where the objective trades off more subjective generation properties, since this seems like a particularly natural use case for the framework.
- The gains from composition itself are not uniformly large. In several settings, a single regularizer is enough to realize most of the improvement over the base sampler, and the composed objective mainly provides a better trade-off between metrics rather than a large absolute gain.
- One other minor concern: objectives without closed-form solutions require iterative mirror ascent and add some inference overhead.
Questions for authors
Thank you for submitting an interesting paper!
One clarifying question: Most of the main experiments use fixed decoding hyperparameters (e.g., , , and equal composition weights). How much of the reported improvement, particularly the pass@1 gains, is still left over when the standard samplers and single-regularizer baselines are individually tuned? I think this would help clarify how much of the gain comes specifically from composition rather than from moving to a better point in the decoding hyperparameter space.
On presentation: I found the main results in Figure 2 somewhat difficult to parse at a glance, especially when comparing the base sampler, individual regularizers, and compositions across settings. The plots are useful for showing trade-offs, but make it harder to see the magnitude of the gains or which method is strongest on a particular metric. To improve presentation and clarity, perhaps the authors could include a compact summary table in the main body (for example, comparing the best standard sampler, single regularizer, and composition for each setting)? This would make the overall empirical picture much easier to follow.
3 comments
| Task | Method | Pass@1 | Pass@4 | Pass@16 |
|---|---|---|---|---|
| MATH500 | Top-k tuned for pass@1 (T=0.10) | 63.0 | 77.2 | 86.0 |
| MATH500 | Top-k tuned for pass@16 (T=0.25) | 62.8 | 82.0 | 92.0 |
| MATH500 | Top-k KL + Diversity (T=0.25) | 65.2 | 84.0 | 92.0 |
| MATH500 | Top-k KL + Diversity (T=0.50) | 64.4 | 81.4 | 90.0 |
| MATH500 | Min-p tuned for pass@1 (T=0.10) | 64.4 | 76.2 | 84.4 |
| MATH500 | Min-p tuned for pass@16 (T=1.00) | 46.2 | 77.8 | 89.6 |
| MATH500 | Min-p JS + Coverage (T=0.50) | 67.2 | 84.6 | 90.2 |
| GPQA Diamond | Top-k tuned for pass@1 (T=0.25) | 24.7 | 48.5 | 75.3 |
| GPQA Diamond | Top-k tuned for pass@16 (T=1.00) | 14.1 | 49.5 | 86.9 |
| GPQA Diamond | Top-k KL + Diversity (T=0.50) | 21.2 | 50.5 | 83.8 |
| GPQA Diamond | Typical tuned for pass@1 (T=0.50) | 20.7 | 57.6 | 80.3 |
| GPQA Diamond | Typical tuned for pass@16 (T=1.00) | 17.2 | 43.4 | 80.8 |
| GPQA Diamond | Typical KL + Coverage + Diversity (T=0.50) | 25.3 | 60.1 | 83.3 |
| IFEval | Top-p tuned for pass@1 (T=0.25) | 52.1 | 67.5 | 75.6 |
| IFEval | Top-p tuned for pass@16 (T=1.00) | 50.3 | 72.3 | 84.3 |
| IFEval | Top-p JS + Entropy (T=0.50) | 53.1 | 70.4 | 80.0 |
| IFEval | η-sampling tuned for pass@1 (T=0.25) | 54.2 | 63.2 | 72.6 |
| IFEval | η-sampling tuned for pass@16 (T=1.00) | 49.0 | 71.3 | 83.2 |
| IFEval | η-sampling KL + Coverage (T=0.50) | 51.9 | 72.3 | 83.4 |
| LiveCodeBench v6 | Top-p tuned for pass@1 (T=0.10) | 58.3 | 62.9 | 67.4 |
| LiveCodeBench v6 | Top-p tuned for pass@16 (T=1.00) | 53.7 | 65.1 | 69.7 |
| LiveCodeBench v6 | Top-p KL + Diversity (T=0.50) | 56.6 | 65.7 | 68.6 |
| LiveCodeBench v6 | Min-p tuned for pass@1 (T=0.10) | 56.6 | 62.3 | 65.7 |
| LiveCodeBench v6 | Min-p tuned for pass@16 (T=1.00) | 54.3 | 64.6 | 67.4 |
| LiveCodeBench v6 | Min-p JS + Diversity (T=0.50) | 54.9 | 65.1 | 69.7 |
Table 3: Comparison of Composable Sampler with temperature-tuned baselines across four benchmarks.
One other minor concern: objectives without closed-form solutions require iterative mirror ascent and add some inference overhead.
We acknowledge that iterative mirror ascent introduces additional inference overhead, with detailed discussion in Section 5.1 and Appendix D.3. We note that an LLM forward pass has , where , , , and denote the number of layers, hidden dimension, context length, and vocabulary size. Each mirror-ascent update has , where is the sampling support size, resulting in for iterations. In our main experiments, we use , with and (e.g., for the default Top-k configuration). Thus, for the regularizers considered in this paper, the additional optimization does not change the overall asymptotic complexity of decoding, does not directly depend on the number of layers and the hidden dimension , and requires no additional model forward passes. The number of iterations also provides a trade-off between computational cost and optimization accuracy, we will further clarify this in the revised manuscript.
I found the main results in Figure 2 somewhat difficult to parse at a glance, especially when comparing the base sampler, individual regularizers, and compositions across settings. The plots are useful for showing trade-offs, but make it harder to see the magnitude of the gains or which method is strongest on a particular metric. To improve presentation and clarity, perhaps the authors could include a compact summary table in the main body? This would make the overall empirical picture much easier to follow.
We agree that a summary table would make it easier to compare the performance of different methods. Figure 2 is mainly intended to illustrate the trade-offs between different sampling methods across tasks. We will include a compact table with simplified results in the main text of the revised manuscript to improve readability and highlight the magnitude of the improvements. We thank the reviewer for this suggestion.
Thank you so much for the new results regarding open-ended generation, baseline tuning, and presentation! This addresses most of my major concerns, and I'm happy to maintain my strong accept recommendation.
We thank the reviewer for the positive assessment and constructive suggestions. We will include additional evaluations of open-ended generation and tuned baselines in the revised manuscript.
We additionally evaluate Qwen2.5-7B on the poetry couplet completion task from FUDGE [1] to extend our evaluation to open-ended generation. Given the first line of the closing couplet of each of 154 Shakespearean sonnets, the model generates the next line. We report pass@1, pass@4, and pass@16 based on the benchmark's criteria, including iambic pentameter and rhyme. As shown in Table 1, our compositional sampler also shows improvements on this task with both base samplers. With Min-p sampling, KL+Diversity improves pass@1 from 9.7 to 16.9 while maintaining pass@16 at 63.0, while KL+Coverage improves pass@16 from 63.0 to 66.2. With Top-p sampling, KL+Coverage improves pass@1 from 12.3 to 16.2 and pass@16 from 65.6 to 68.2. We will include these additional results on constrained open-ended generation in the revised manuscript.
Table 1: Evaluation on open-ended generation using Qwen2.5-7B
[1] Yang, Kevin, and Dan Klein. "FUDGE: Controlled text generation with future discriminators." Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021
We agree that a single regularizer can account for much of the improvement in a particular metric, while different regularizers can provide gains on different metrics. Composition can combine these complementary benefits and achieve a better balance across different metrics using a single configuration.
We tuned the hyperparameters of standard samplers, individual regularizers, and their composition on a small validation set using Optuna, and evaluated the selected configurations on the MATH500 test set. As shown in Table 2, our composition achieves 65.2 pass@1, compared with 59.8 for tuned Top-k and 64.8 for KL, the best-performing tuned single regularizer. Although the pass@1 gain over KL is relatively small, our composition also improves pass@4 from 79.6 to 81.6 and pass@16 from 86.6 to 89.4, showing a better balance across different sampling budgets. We also report additional temperature-tuning results across four benchmarks in Table 3. Across all eight configurations, our composable sampler, without task-specific temperature tuning, improves pass@16 over the pass@1-tuned baseline and pass@1 over the pass@16-tuned baseline. For example, on MATH500 with min-p, the pass@1-tuned baseline achieves (64.4, 84.4) and the pass@16-tuned baseline achieves (46.2, 89.6), whereas our composition achieves (67.2, 90.2) in a single configuration. We will include these additional results and tuning details in the revised manuscript and release the hyperparameter tuning interface.
Table 2: Comparison of tuned standard samplers, individual regularizers, and composition on MATH using Qwen2.5-7B