- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper takes a deep dive into the problem of LLM decoding, specifically focused on exploring the optimal temperature needed for pass@k maximization. The authors provide significant theoretical evidence that their tilted temperature proposition is optimal for pass@k optimization subject to constraints that prevent two well-known failure modes in decoding (Dirac sampling on one hand and uniform sampling on the other). The optimal temperature is difficult to estimate so the authors provide two practical implementations in different regimes. Empirical results of deploying this optimal temperature strategy on a coding task show improvement for a few coding LLM variants.
Critical strengths
Strength 1: theoretical backed decoding temperature optimality when the pass@k objective is subject to practical constraints to avoid extreme failure modes.
Strength 2: empirical results that back the theoretical analysis and specifically the practical algorithms proposed for computing the optimal temperature.
Critical weaknesses
The reviewer does not see any major weaknesses in this theoretical paper that support rejection.
Questions for authors
-
The authors do not provide a computational analysis of the proposed temperature estimation strategy as compared to the non-optimal baselines (e.g. Fixed T and AdaptT). It is important to know what the extra computational cost will be if the optimal temperature is computed regularly. If this step significantly slows down decoding, this will be an issue for practical use.
-
The authors mention that CTD tracks other temperature strategies; however, the results in Table 1 show that CTD can be much worse even as compared to Fixed T. Can the authors elaborate on why this is?
We thank the reviewer for the positive assessment and the two practical questions. We also refer the reviewer to the general response for the corrected Table 1.
Computational cost: Fixed-T, AdapT and CTD have identical generation cost (one forward pass per token). CTD's calibration is a one-off offline step on the training problems: sampling a small calibration pool, running the verifier, and a scalar OLS fit. TMAD-Gated is more expensive than these baselines. Its overhead comes from the short lookahead, while the continuation head itself is a low-rank bilinear map with negligible cost. The gate confines this overhead to a subset of positions: tokens where the model is uncertain or that are structurally critical, such as the beginning of a new logical statement in code. Elsewhere, decoding uses conservative sampling at the cost of plain temperature sampling. With an active fraction α of positions, c refined candidates and lookahead depth L, this amounts to 1+αcL single-token forward passes per generated token (each lookahead step reuses the cached prefix), plus a copy of the KV cache per refined candidate; c and L directly trade cost for accuracy.
Why can CTD be worse than Fixed-T: This is consistent with the theory: Theorem 7 states that CTD should track the best temperature, not beat it, up to the dropped correction term and the calibration error ε. Unlike CTD, Fixed-T is tuned directly to maximize validation pass@5. CTD instead selects its exponent through entropy matching, using the fitted calibration relationship as a certificate. This entropy-matched exponent need not coincide with the pass@k-tuned temperature, especially under imperfect calibration or residual continuation effects. Thus Fixed-T can outperform CTD in some settings. We have clarified this distinction in the revised text.