- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents Lookahead-based Rescoring (LoRe), a decoder that selects the next token by using a d-step greedy lookahead log joint probability to overcome the problem of low-probability first-tokens. During decoding, LoRe is applied with a tunable probability alpha and falls back to the default decoding method (min-p or top-p) with probability 1-alpha. The authors showed that LoRe could be used with min-p to produce creative generations that are generally better than untuned min-p decoding in terms of creativity, fluency, coherence and rare multi-word expressions.
Critical strengths
- The authors showed that while models have strong internal representations of idioms and their literal counterparts via strong separability of their pooled hidden states, the models' generated distribution of idioms is significantly lower than the estimated training distribution.
- A first-token decoding bottleneck is demonstrated, where the first token of idiom sequences are unlikely but which accumulates greater joint probabilities if allowed to realize.
- LoRe's positive effects with min-p is shown in Figure 2 (unfortunately LoRe's effect is inconclusive with top-p in Figure 4)
- A range of metrics is used for evaluation of creative generation (creativity, fluency and coherence)
Critical weaknesses
- Insufficient baseline comparisons: Since LoRe is primarily targeted at creative generation, it should be compared to more baselines that also address creative generation besides top-p and min-p. Examples include Mirostat (Basu et al, 2021), Improving open-ended text generation via adaptive decoding (Zhu et al 2024), etc.
- The tasks for non-creative generation are unclear and insufficient: Evaluation of non-creative generation is performed via prompting the models with the <BOS> token without task-specific problems. It is unclear how this enables robust assessment of the qualities of non-creative generation, including those grounded in reasoning domains. An acceptable comparison should at least preserve reasonable accuracy levels.
- The current format narrowly frames the paper as an improvement over min-p on creative generation. Furthermore, despite L209 proposing LoRe as complementary to existing truncation-methods, LoRe does not practically improve over top-p as shown in Figure 4.
- By tuning LoRe's alpha and being able to access optimal alphas across models, there seems to be an unfair advantage against the chosen top-p and min-p baselines whose hyperparameters should also be tuned rather than fixed.
- Weak LLM and human evaluations: Only one LLM judge is used; more judges are needed to reduce bias and measure their agreement rates. Human evaluation of 10 stories for each activation probability is also too small a sample size.
- Although the results show that LoRe has a higher win-rate against min-p on coherence (L335) and the evaluation criteria for coherence is included in the prompt (L837), it is unclear how LoRe leads to greater coherence without actual annotated generated examples that can be related to the mechanics of LoRe. On the other hand, the winner for coherence is inconclusive between LoRe and top-p (L943), despite the paper's title on improving coherence.
- Lack of analysis of false positives: Does this method produce continuations that are similar to idioms in terms of the likelihood landscape but are degenerate or non-meaningful?
- Lack of efficiency analysis: Could you perform efficiency analysis on LoRe versus the other baseline methods?
Questions for authors
Please refer to questions, clarifications and feedback in weaknesses.
Baselines. We have added two baselines matched to LoRe’s diversity level. The first is min-p at T=0.85 (distinct-n 0.862 vs. 0.870; self-BLEU 0.056 vs. 0.048). Against it, LoRe still wins on creativity 74% of the time. The second is contrastive search (k=4, penalty=0.6; distinct-n 0.865). It is the most fluent and coherent baseline but the least creative (15–25% wins vs. 53–60% losses for creativity) and the weakest on MWEs, so LoRe offers a different trade-off. We have not yet run Mirostat or adaptive decoding (Zhu et al., 2024). We will add both in the revision.
Non-creative tasks. We agree that <BOS>-only prompting is not a task-based evaluation. We will rename this setting “unconditional (open) generation”, since it tests generation without a prompt rather than non-creative tasks. On a new set of 100 idiom-free news and Wikipedia openings, across 4 models, the LLM judges rated LoRe as fluent and coherent as min-p up to α = 0.5, with LoRe ahead or behind by at most 4 points. At α = 0.7 LoRe fell behind (fluency 22% wins vs 44% losses). LoRe did not raise creativity or rare-MWE use in these contexts. We recommend low α for non-creative use.
Framing and top-p (L209, Fig. 4). We agree. LoRe does not give a practical improvement over top-p, if added to top-p itself. However, min-p+LoRe has an advantage in both creativity and coherence over top-p.
Hyperparameter fairness. The diversity-matched baselines above are tuned specifically to be comparable to LoRe. On LoRe’s side, we use the same α = 0.6 for all models rather than choosing α separately for each model. In the blind rerun, LoRe’s creativity advantage over min-p averages 81% across the α values tested, and coherence is best at α=0.5–0.7 (65%). We also swept lookahead depth (d ∈ {1, 2, 4, 6}); the differences between depths fall within the confidence intervals, so the results do not depend on choosing a particular depth. We have not yet tuned the baselines’ own hyperparameters across a full grid. We will add these sweeps, along with per-α blind win rates and confidence intervals, in the revision.
LLM and human evaluation. We have re-judged all comparisons with three judges from different model families (Claude Sonnet 5.5, GPT-5.6 and Qwen2.5-72B), using labels that hide the decoding method. The original prompts exposed the decoding parameter. The results hold. LoRe averages an 81% creativity win rate against min-p. At α=0.5–0.7 it reaches 65% on coherence and roughly ties on fluency. We have not yet run a larger human study. We will run one with substantially more stories per α and report Cohen’s κ between the human annotators and each judge.
How LoRe improves coherence. A new ablation addresses the mechanism. At the same α=0.5 with the same candidates, choosing randomly among the candidates gives the worst coherence of all variants (25–28% wins vs. 47–50% losses against min-p). With lookahead (d=4), the same set of candidates beats min-p on coherence. The lookahead score favors candidates whose continuations the model predicts confidently, which keeps the exploration from drifting off course. We agree that coherence against top-p is inconclusive, and we will soften the claim in the title and text accordingly. We have not yet prepared annotated example generations. We will add examples to the appendix that link specific lookahead decisions to their effect on coherence.
False positives. This is a good point. Fluency stays roughly tied with min-p across judges, which suggests that degenerate selections are not widespread. However, we have not yet analyzed directly the steps where lookahead changes the chosen token. We will annotate a sample of these steps as meaningful MWE, neutral, or degenerate, and report the rates.
Efficiency. Each activated step adds k·d forward passes. The k rollouts run in parallel as one batch with a KV cache, so an activated step costs d sequential batched passes. The expected overhead is α·d batched passes per generated token, which equals 2.4 at α=0.6 and d=4. We have not yet measured wall-clock throughput. We will report tokens per second for LoRe and each baseline in the revision.