acceptodds
Withdrawn from ICLR 2027

Lost in the Long Tail: Improving Creativity and Coherence in Language Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

Decoding strategies strongly shape the behavior of language models in open-ended generation. Prior work has focused on truncating the distribution over next tokens in order to remove noise. However, do improbable tokens always constitute noise? We examine a phenomenon we term token myopia, in which truncation decoding methods systematically under-generate valid multi-token lexical units because their initial token receives low probability despite being followed by a highly probable continuation. Using idioms and collocations as a measurable testbed, we show that language models often encode idiomatic structure internally yet fail to realize these expressions in free generation due to this first-token bottleneck. To address this limitation, we propose Lookahead-based Rescoring (LoRe), a short-horizon sequence-scoring decoder that rescales candidate next tokens using d-step greedy lookahead continuation scores. To balance quality and diversity, we combine this approach with a stochastic activation mechanism that applies lookahead at a tunable rate. LoRe is complementary to existing truncation-based methods such as min-p sampling, allowing the model to generate more long-tail expressions without altering the underlying model. Across creative generation tasks, LoRe increases multi-word expression usage by 50% on average relative to min- sampling. Furthermore, it outperforms the baseline in 86% of pairwise evaluations on average in creativity, while preserving fluency and coherence. These results suggest that many decoding failures do not stem from deficiencies in model knowledge, but from myopic token-level decision making, and that applying a short-horizon lookahead can better align generation with the latent linguistic structures already represented within language models.

Primary area:
generative models
Keywords:
Creative generation, decoding strategy, large language models, token myopia

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_7KyvOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents Lookahead-based Rescoring (LoRe), a decoder that selects the next token by using a d-step greedy lookahead log joint probability to overcome the problem of low-probability first-tokens. During decoding, LoRe is applied with a tunable probability alpha and falls back to the default decoding method (min-p or top-p) with probability 1-alpha. The authors showed that LoRe could be used with min-p to produce creative generations that are generally better than untuned min-p decoding in terms of creativity, fluency, coherence and rare multi-word expressions.

Critical strengths
  • The authors showed that while models have strong internal representations of idioms and their literal counterparts via strong separability of their pooled hidden states, the models' generated distribution of idioms is significantly lower than the estimated training distribution.
  • A first-token decoding bottleneck is demonstrated, where the first token of idiom sequences are unlikely but which accumulates greater joint probabilities if allowed to realize.
  • LoRe's positive effects with min-p is shown in Figure 2 (unfortunately LoRe's effect is inconclusive with top-p in Figure 4)
  • A range of metrics is used for evaluation of creative generation (creativity, fluency and coherence)
Critical weaknesses
  • Insufficient baseline comparisons: Since LoRe is primarily targeted at creative generation, it should be compared to more baselines that also address creative generation besides top-p and min-p. Examples include Mirostat (Basu et al, 2021), Improving open-ended text generation via adaptive decoding (Zhu et al 2024), etc.
  • The tasks for non-creative generation are unclear and insufficient: Evaluation of non-creative generation is performed via prompting the models with the <BOS> token without task-specific problems. It is unclear how this enables robust assessment of the qualities of non-creative generation, including those grounded in reasoning domains. An acceptable comparison should at least preserve reasonable accuracy levels.
  • The current format narrowly frames the paper as an improvement over min-p on creative generation. Furthermore, despite L209 proposing LoRe as complementary to existing truncation-methods, LoRe does not practically improve over top-p as shown in Figure 4.
  • By tuning LoRe's alpha and being able to access optimal alphas across models, there seems to be an unfair advantage against the chosen top-p and min-p baselines whose hyperparameters should also be tuned rather than fixed.
  • Weak LLM and human evaluations: Only one LLM judge is used; more judges are needed to reduce bias and measure their agreement rates. Human evaluation of 10 stories for each activation probability is also too small a sample size.
  • Although the results show that LoRe has a higher win-rate against min-p on coherence (L335) and the evaluation criteria for coherence is included in the prompt (L837), it is unclear how LoRe leads to greater coherence without actual annotated generated examples that can be related to the mechanics of LoRe. On the other hand, the winner for coherence is inconclusive between LoRe and top-p (L943), despite the paper's title on improving coherence.
  • Lack of analysis of false positives: Does this method produce continuations that are similar to idioms in terms of the likelihood landscape but are degenerate or non-meaningful?
  • Lack of efficiency analysis: Could you perform efficiency analysis on LoRe versus the other baseline methods?
Questions for authors

Please refer to questions, clarifications and feedback in weaknesses.

1 comment

Baselines. We have added two baselines matched to LoRe’s diversity level. The first is min-p at T=0.85 (distinct-n 0.862 vs. 0.870; self-BLEU 0.056 vs. 0.048). Against it, LoRe still wins on creativity 74% of the time. The second is contrastive search (k=4, penalty=0.6; distinct-n 0.865). It is the most fluent and coherent baseline but the least creative (15–25% wins vs. 53–60% losses for creativity) and the weakest on MWEs, so LoRe offers a different trade-off. We have not yet run Mirostat or adaptive decoding (Zhu et al., 2024). We will add both in the revision.

Non-creative tasks. We agree that <BOS>-only prompting is not a task-based evaluation. We will rename this setting “unconditional (open) generation”, since it tests generation without a prompt rather than non-creative tasks. On a new set of 100 idiom-free news and Wikipedia openings, across 4 models, the LLM judges rated LoRe as fluent and coherent as min-p up to α = 0.5, with LoRe ahead or behind by at most 4 points. At α = 0.7 LoRe fell behind (fluency 22% wins vs 44% losses). LoRe did not raise creativity or rare-MWE use in these contexts. We recommend low α for non-creative use.

Framing and top-p (L209, Fig. 4). We agree. LoRe does not give a practical improvement over top-p, if added to top-p itself. However, min-p+LoRe has an advantage in both creativity and coherence over top-p.

Hyperparameter fairness. The diversity-matched baselines above are tuned specifically to be comparable to LoRe. On LoRe’s side, we use the same α = 0.6 for all models rather than choosing α separately for each model. In the blind rerun, LoRe’s creativity advantage over min-p averages 81% across the α values tested, and coherence is best at α=0.5–0.7 (65%). We also swept lookahead depth (d ∈ {1, 2, 4, 6}); the differences between depths fall within the confidence intervals, so the results do not depend on choosing a particular depth. We have not yet tuned the baselines’ own hyperparameters across a full grid. We will add these sweeps, along with per-α blind win rates and confidence intervals, in the revision.

LLM and human evaluation. We have re-judged all comparisons with three judges from different model families (Claude Sonnet 5.5, GPT-5.6 and Qwen2.5-72B), using labels that hide the decoding method. The original prompts exposed the decoding parameter. The results hold. LoRe averages an 81% creativity win rate against min-p. At α=0.5–0.7 it reaches 65% on coherence and roughly ties on fluency. We have not yet run a larger human study. We will run one with substantially more stories per α and report Cohen’s κ between the human annotators and each judge.

How LoRe improves coherence. A new ablation addresses the mechanism. At the same α=0.5 with the same candidates, choosing randomly among the candidates gives the worst coherence of all variants (25–28% wins vs. 47–50% losses against min-p). With lookahead (d=4), the same set of candidates beats min-p on coherence. The lookahead score favors candidates whose continuations the model predicts confidently, which keeps the exploration from drifting off course. We agree that coherence against top-p is inconclusive, and we will soften the claim in the title and text accordingly. We have not yet prepared annotated example generations. We will add examples to the appendix that link specific lookahead decisions to their effect on coherence.

False positives. This is a good point. Fluency stays roughly tied with min-p across judges, which suggests that degenerate selections are not widespread. However, we have not yet analyzed directly the steps where lookahead changes the chosen token. We will annotate a sample of these steps as meaningful MWE, neutral, or degenerate, and report the rates.

Efficiency. Each activated step adds k·d forward passes. The k rollouts run in parallel as one batch with a KV cache, so an activated step costs d sequential batched passes. The expected overhead is α·d batched passes per generated token, which equals 2.4 at α=0.6 and d=4. We have not yet measured wall-clock throughput. We will report tokens per second for LoRe and each baseline in the revision.

Reviewer_Y6iMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that LLMs rarely produce idioms and other multi-word expressions in free generation, even though these appear in pretraining data. The authors call this "token myopia", meaning the first token of an idiom has low probability so truncation sampling cuts it, even if the rest of the idiom is very likely. They support this with idiom counts in generations and with probing and probability analysis. They then propose LoRe, which at each step (with probability alpha) scores candidate tokens by their own log-prob plus the log-prob of a short greedy rollout, and otherwise falls back to min-p. They report large gains in creativity and rare MWE counts over min-p, with similar fluency and somewhat better coherence.

Critical strengths

The question the paper asks is a good one. The idea that truncation methods can throw away a low-probability token that starts a very likely phrase is intuitive, and idioms are an easy way to measure it. The first-token probability plots make the point easy to see. The method is simple and training-free, and it can be used on top of min-p, so it could be useful for creative writing if the results hold.

Critical weaknesses

The motivation numbers in Section 2 do not seem consistent to me. The paper says idioms make up 0.31% of tokens in the Pile and also that this equals about 9.3 million tokens. The Pile has a few hundred billion tokens, so 9.3 million is closer to 0.003%. If the smaller number is right, the expected count in 562k generated tokens is around 17, not 1630, and the observed 26 to 36 idiom tokens for most models would not be a deficit at all. Since the whole "LLMs under-generate idioms" claim rests on this comparison, this needs to be fixed or explained. Also, Section 2.3 samples with p = 1.0, so there is no truncation there, and even if a deficit existed it could not be blamed on truncation cutting the first token.

My bigger concern is the evaluation of LoRe. Everything is judged by one small, older model (Claude-3-Haiku), and the creativity, fluency and coherence prompts include "Story A (p=X)", which may tell the judge the decoding setting. The MWE prompt rewards rare, non-clichéd phrases and even lists "kick the bucket" as an overused idiom not to count, though this is the paper's own motivating example. The idiom count from Section 2 is never reported for LoRe. The human check is only 10 stories per alpha with two annotators, and human to LLM agreement for fluency (64%) and coherence (62%) is not far from chance in a pairwise setup. There are no automatic metrics like diversity or MAUVE, and no comparison to other lookahead methods such as beam search or contrastive search, or even to min-p at a higher temperature. So it is hard to tell if the gains come from lookahead itself or just from sampling with more diversity.

I also think a relevant prior work, Top-H decoding (Potraghloo et al., 2025), which also tries to balance creativity and coherence, is missing from the paper.

Questions for authors
  1. Please check the 0.31% figure against the 9.3M token count. How does this change the claim in Section 2.3?
  2. Was the "(p=X)" label in the judge prompts filled with the real decoding parameter? If yes, please rerun with blind labels.
  3. What is k in the experiments, and what is the extra compute of LoRe (k times d forward passes per activated step) at the alpha values you recommend?

As a suggestion, a comparison with min-p at higher temperature or contrastive search, tuned to a similar diversity level, would make the results much more convincing.

Minor: Section 3 says nucleus sampling is the fallback but Algorithm 1 says min-p, and the text says min-p candidates but Algorithm 1 uses top-k. Section 2.3 mentions "all six models" but only five are listed, and the "highest idiom fraction is 3 × 10^-5" does not match Table 1 (0.000064). For the probe, the literal sentences in Table 2 look quite different in style from the idiomatic ones, so perfect accuracy may come from surface differences and not from idiomaticity.

1 comment

Section 2 statistics (W1, Q1). The reviewer is right. The §2.2 ratios were computed per document, not per token, and the 9.3M figure belongs to RedPajama, not the Pile. On a token basis, idioms make up about 0.0005% of the Pile. The corpus-level comparison in §2.3 therefore does not show under-generation, and we remove it. We instead base the claim on direct, context-conditioned evidence of token myopia. In contexts that cue an idiom, the first idiom token has a mean probability of only 0.05–0.07, and the following tokens become highly likely once it is chosen (§2.5). Standard truncation removes this first token before its continuation can be realized: in 47.3% of contexts under min-p (p=0.02) and 29.6% under top-p (p=0.95), pooled over five models, with consistent rates across models (min-p 43.5–51.1%). As the reviewer notes, §2.3 used p=1.0, so truncation cannot explain any effect there. We now treat these as two separate effects. Low probability on the idiom’s first token is present even without truncation. Truncation then removes that token outright.

Judge labels and judge choice (W2, Q2). Yes, the labels in the original prompts contained the real decoding parameter, so the judge could see which story came from which setting. We have rerun every comparison with fully blind labels (“Story A” / “Story B”, with no decoding information), using three current judges from different model families (Claude Sonnet 5.5, GPT-5.6 and Qwen2.5-72B) in place of Claude-3-Haiku. All LoRe results reported in this response come from the blind rerun. The advantages remain. LoRe averages an 81% creativity win rate against min-p. At α=0.5–0.7 it reaches 65% on coherence and roughly ties on fluency. LoRe generations contain 3.0 rare MWEs per story on average, compared with 2.1 for min-p (difference CI [0.48, 1.32]).

MWE metric and the §2 idiom count. We changed the example. For the idiom pairs mentioned in section 2, the highest count is LoRe at α=\alpha= 0.7 with 0.03 idioms per story, compared to min-p baseline with 0.01 idioms per story.

Human evaluation. To obtain more reliable evaluations, we moved to three blind judges. We have not yet run a larger human study. We will run one with substantially more pairs and annotators and report Cohen’s κ between the human annotators and each judge.

Lookahead or just diversity? We ran the comparisons the reviewer suggested.

  • Diversity: LoRe is slightly less diverse than standard min-p (distinct-n 0.870 vs. 0.878; self-BLEU 0.048 vs. 0.041), so its gains do not come from extra diversity. Against min-p at T=0.85 (distinct-n 0.862, self-BLEU 0.056), LoRe still wins on creativity 74% of the time.

  • Contrastive search: At a similar diversity level (k=4, penalty=0.6; distinct-n 0.865), contrastive search is the most fluent and coherent baseline but the least creative (15–25% wins vs. 53–60% losses) and the weakest on MWEs.

  • Isolating the lookahead signal: At the same α=0.5 with the same candidates, taking the argmax with no lookahead (d=0) loses to min-p on every metric. Choosing randomly among the candidates is creative but the worst on fluency and coherence. Only lookahead (d=4) beats min-p on both creativity and coherence.

  • Not yet run. We have not yet run beam search, Top-H or MAUVE. We will add all three in the revision.

  • Top-H. Thank you for the pointer. Top-H (Potraghloo et al., 2025) adapts truncation by bounding the entropy of the kept distribution, but like min-p it ranks tokens by next-token probability alone, so it can still drop a low-probability first token of a likely multi-token continuation. LoRe rescores candidates by lookahead likelihood and is complementary, so it can run on top of Top-H. We will cite and compare against it.

Q3 (k and compute). Here k is the number of candidates min-p keeps at each step, so it varies by step. Each activated step adds k·d forward passes. The k rollouts run in parallel as one batch with a KV cache, so an activated step costs d sequential batched passes. The expected overhead is therefore α·d batched forward passes per generated token, which equals 2.4 at our recommended α=0.6 and d=4. We have not yet measured wall-clock throughput. We will report it against the baselines in the revision.

Minor points. The fallback is min-p throughout, and we will fix §3 and make Algorithm 1’s candidate selection match the text. We will correct “six models” to five. On the probe: each pair uses the same idiom in both its literal and idiomatic sentence, so idiom identity alone cannot predict the label. However, we agree that differences in style may contribute. We have not yet run the probe with a split grouped by idiom or with style-matched controls. We will run both and report the results.

Reviewer_kWR5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work study why llm under-generate idioms and other multi-word expressions. They propose LoRe to switch between ordinary next token prediction or probed lookahead prediction. LoRe scores candidate tokens and short continuations, and it also swap to min-p sampling under inactive lookahead. They validated it on Llama-3.1-8B, Gemma-2-9B, and Pythia-12B base models with claude 3 as judge. It also uses claude 3 as judge, and a small human check is used in evaluation too. Through the experiments, an increase of rare expression, creativity, and coherence rating is observed.

Critical strengths
  1. A comprehensive view across corpus count, probes, etc. that provide insight to the first token bottleneck. This combination shows where phrase generation fail and motivate the solution for decoding time algorithm.
Critical weaknesses
  1. Eval metric is not sufficiently justified and is quite vague. LLM judge with few small human judge leaves many uncertainties. No confidence interval/error bar are reported, and a strong correlation between human judge and LLM judge need more data.
  2. Motivation are very weakly justified and experiment design does not align with the motivation. The reported gain is not attribute directly to the algorithm combination (lookahead and sampling are not independently evaluated).
Questions for authors
  1. This is for motivation of this work. Why more idioms or unusual expression is better?
  2. How do you know the llm judge work correctly? You need a large amount of correlation study to justify this.
  3. LoRe change both information for selection and selection rule at the same time, so it can't tell whether it is the benefit of lookahead or benefit of selection. For example, can a simple LLM generation temperature change in the beginning perform similar too?
1 comment

W1, Q2. We have re-judged all comparisons with labels that hide the decoding method, using three judges from different model families (Claude Sonnet 5.5, GPT-5.6 and Qwen2.5-72B). Our conclusions hold under all three. LoRe averages an 81% creativity win rate against min-p. At α=0.5–0.7 it reaches 65% on coherence and roughly ties on fluency. LoRe generations contain 3.0 rare MWEs per story on average, compared with 2.1 for min-p (difference CI [0.48, 1.32]). In the new ablations, all three judges agree on which method wins on every metric. We have not yet run a larger human study. We will run one, with substantially more pairs per metric and more annotators, and report Cohen’s κ between the human annotators and each judge.

W2, Q1. We do not claim that more idioms are better in themselves. Lack of figurative and idiomatic language is one of the documented ways LLM text differs from human writing (Reinhart et al., 2025; Muñoz-Ortiz et al., 2024). Idioms are a measurable case of a broader failure, and our MWE metric counts only uses that fit naturally in context. We have also strengthened the motivating evidence. In contexts that cue an idiom, the idiom’s first token has a mean probability of only 0.05–0.07. Standard truncation removes it in 47.3% of contexts under min-p and 29.6% under top-p, even though the rest of the idiom becomes highly likely once that first token is chosen. LoRe gives these expressions back to the model without hurting fluency or coherence.

W2, Q3 Separating the selection rule from the lookahead signal: We compared three variants at the same α=0.5 with the same candidates and their only difference is the way the token is chosen. Taking the argmax over the candidates with no lookahead (d=0) loses to min-p on every metric (creativity: 20–26% wins vs. 52–54% losses). Choosing randomly among the candidates gives the most creativity and MWEs, but the worst fluency (23–34% wins vs. 46–47% losses) and coherence (25–28% vs. 47–50%). Only lookahead (d=4) beats min-p on both creativity and coherence. The changed selection rule alone therefore does not produce the gains. Lookahead acts as a sharpening step that keeps the creativity of exploring other candidates while restoring coherence.

Diversity-matched baselines. LoRe is slightly less diverse than standard min-p (distinct-n 0.870 vs. 0.878; self-BLEU 0.048 vs. 0.041). Its gains therefore cannot come from extra diversity, and raising the temperature would only push the baseline further from LoRe on diversity. Against min-p at T=0.85 (distinct-n 0.862, self-BLEU 0.056), LoRe still wins on creativity 74% of the time. Contrastive search at a similar diversity level (distinct-n 0.865) is the most fluent and coherent baseline but the least creative (creativity: 15–25% wins vs. 53–60% losses). This suggests LoRe offers a different trade-off from contrastive search, not the same effect. We tested global temperature, not a temperature schedule that changes over the course of generation, such as a higher temperature at the beginning. We have not yet run that comparison and will include it in the revision.

Reviewer_7KMkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper identifies the issue of under-generation of idiomatic expressions in LLMs across different open-source models. They attribute this to the low probability of the first token in an idiom, but high probability after the first token, which gets truncated with other decoding methods. To prevent this and allow generation of idiomatic expressions, the authors propose a decoding method: LoRe. This method rescores the candidate next tokens by appending each candidate to the current context and evaluating a short greedy rollout.

The authors test this on two sets of continuations and show that the output is more creative, coherent, and has more rare words compared to min-p and top-p sampling methods as baselines. They also show that their method does not lead to a loss of fluency etc for general text.

Critical strengths

Strengths:

  1. The paper proposes a decoding time technique that is easy to integrate with existing methods
  2. The paper shows that their method performs better than existing decoding techniques
  3. The paper includes validation attempts
Critical weaknesses

Weaknesses:

  1. The proposed method (in Section 3.1) seems that it would be useful for any sets of continuations that score higher on more than a single token rollout, compared to be specific to idioms.

  2. In section 2.3, the authors only test open-generation, and they do not test on any context, which is how most models are generally

  3. The classification probe does not stratify based on the idioms; also the context under which the literal and the idiomatic interpretations are used is different, so it is possible that the model detects something more than the idiomatic usage of the phrase. (Section 2.4)

  4. For 2.5, the paper does not specify the number of instances tested / number of instances with 2 tokens (in figure 1)

  5. The experimental set up has a weak human evaluation [only on 10 instances]

  6. Results from Section 5 are not repeated in the same set up as Section 2; we don’t see if the decoding method increases the idiom presence in the generated text; the results are only one metrics like creativity, fluency etc

  7. In Section 4: the non-creative prompts should maybe be named as “open-generation” since it is not conditioned on any text rather than the text being non-creative

  8. Missing computational costs associated with the proposed method and how much overhead this adds to the inference

  9. The claim in the paper about the approach being for creative and open-ended generation seems overstated. The method seems like an extension of methods like NeuroLogic (cited in line #447), which also scores candidates by look ahed probabilities, so the novelty of the proposed approach is unclear.

Inconsistencies:

(a) Line #124 says the highest observed idiom fraction is 3 x 10^-5 but in Table 1 the highest is 6.4 x 10^-5.

(b) In Line #134 the paper says six models, but there are only five models reported in Table 1

(c) The method description is inconsistent - in line 200 the paper says the fallback is nucleus sampling, in the introduction it says the fall back is min-p sampling also in algorithm 1 line 12 it says min-p

Questions for authors
  1. How many tokens does the pile dataset have? What tokenizer do you use for Section 2.2? How do you detect idioms in the output (for Table 1)?

  2. How often are idiom-initial tokens removed/truncated? Did you look at the probability distribution? (section 2.3)

  3. For section 2.4, how do you create a set of 500 prompts/instances, when you have 188 selected idioms? Do you repeat them, if so, then how? And then can your train-test split have the same idiom but in different contexts since you only split by class and not idioms?

  4. For section 2.5, how many continuations did you sample for the literal baseline?

  5. Can you report performance in Section 5 per-dataset? Can you also give examples?

  6. What are the generation parameters for the creative prompts reported under Section 5?

  7. Does your method work on instruction tuned models / larger models / other genres?

  8. What are extra computational costs associated with this set up when doing look ahead rollouts?

  9. Did you vary depth=4 when doing lookahead?

1 comment

W1. We agree, and see this as a strength. Token myopia affects any continuation whose first token is unlikely but whose multi-token continuation is likely. Idioms are our measurable testbed, not the only target. We will revise the introduction's framing.

W2, Q2. We replace the conditional comparison with context-conditioned evidence. In the 188 idiom-cueing contexts of §2.5, the first idiom token has mean probability 0.05–0.07. Truncation removes it in 47.3% of contexts under min-p (p=0.02) and 29.6% under top-p (p=0.95), pooled over five models; min-p rates are consistent across models (43.5–51.1%). We will add per-model rates and first-token probability distributions to the appendix.

W3. The 500 pairs use EPIE's multiple usage examples per idiom, so an idiom can appear in both train and test. However, each pair uses the same idiom in its literal and idiomatic sentence, so idiom identity cannot predict the label. We will add an idiom-grouped split and controls for context and style differences.

W4, Q4. Fig. 1 panels contain n=50, 64, 37, 27 idioms for lengths 2–5; the rest have length 1 or >5, which we will state in the caption. We sampled one literal continuation per context and will state this in §2.5.

W5. We agree 10 stories per α\alpha is too few. We re-judged all comparisons blind to decoding method with three judges from different model families (Claude Sonnet 5.5, GPT-5.6, Qwen2.5-72B). LoRe keeps an 81% average creativity win rate; at α\alpha=0.5–0.7 it reaches 65% on coherence and roughly ties on fluency. We will add a larger human study, reporting Cohen's κ between annotators and each judge.

W6. Under blind judging, LoRe stories contain 3.0 rare MWEs on average vs. 2.1 for min-p (difference CI [0.48, 1.32]). For the §2 idioms, the highest rate is LoRe at α=\alpha= 0.7 with 0.03 idioms per story, vs. 0.01 for min-p.

W7. We will rename this setting “unconditional (open) generation”, since it tests generation without a prompt rather than non-creative tasks. On a new set of 100 idiom-free news and Wikipedia openings, across 4 models, the LLM judges rated LoRe as fluent and coherent as min-p up to α = 0.5, with LoRe ahead or behind by at most 4 points. At α = 0.7 LoRe fell behind (fluency 22% wins vs 44% losses). LoRe did not raise creativity or rare-MWE use in these contexts. We recommend low α for non-creative use.

W8. Each activated step adds k·d forward passes, but the k rollouts run as one KV-cached batch, so it costs d sequential batched passes. Expected overhead is α·d batched passes per token: 2.4 at our recommended α=0.6, d=4. We will add measured tokens/second against the baselines.

W9. NeuroLogic A*esque uses lookahead to estimate whether external lexical constraints will be satisfied. LoRe has no constraints: it uses the model's own short-horizon likelihood, activates stochastically, and plugs into truncation samplers. New ablations show the lookahead signal is what helps, not merely a change in token selection. At α=0.5, argmax over the same candidates (d=0) loses to min-p on every metric (creativity: 20–26% wins vs. 52–54% losses). Random choice among the min-p candidates is creative but worst on fluency and coherence. Only lookahead beats min-p on both creativity and coherence. LoRe also wins on creativity (74%) against min-p at T=0.85, which is slightly less diverse (distinct-n 0.862 vs. 0.870; self-BLEU 0.056 vs. 0.048). We will tone down novelty claims and expand related work.

Q1 (Pile size, tokenizer, idiom detection). The Pile has ~300B tokens under the GPT-NeoX tokenizer and ~383B under the Llama-2 tokenizer used by infini-gram, which our §2.2 counts use. Idioms in outputs (Table 1) were detected by exact string matching against the 188 idioms.

Q5 (per-dataset results, examples). We will add per-dataset (BookMIA, Gutenberg) win rates and annotated example generations to the appendix.

Q6 (generation parameters). Creative prompts use 30-token prefixes and up to 256 generated tokens. Min-p: p=0.02, T=1.0. Top-p: p=0.95, T=1.0. LoRe: d=4, k = #min-p candidates (we will fix this inconsistency in the revision), α\alpha ∈ {0.1, …, 0.8}. We will consolidate these into one table.

Q7 (instruct / larger models / other genres). We used base models deliberately, to study free generation without instruction-tuning effects. We will add instruction-tuned models, LoRe on models above 12B, and non-fiction genres in the revision.

Q9 (depth). We swept d ∈ {1, 2, 4, 6}. d=4 is best in 6 of 8 judge × metric cells, though differences fall within CIs. Gains do not grow with depth: d=6 matches d=4 on diversity (distinct-n 0.869 vs. 0.870) without improving quality, supporting d≈4.

Inconsistencies. (b) Corrected to five models. (c) The fallback is min-p throughout; we will fix line 200 and align Algorithm 1's candidate selection with the text. The §2 Qwen model was Qwen2.5-32B; we will correct this throughout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.