- Rating
- 4
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes DRY, a novel decoding algorithm for LLMs that avoid repetitions in a principled way. DRY reduces repetition by penalizing a token if the token reproduces a suffix that has already appeared in the context and if the suffix is of at least certain length L, which is a hyperparameter. While simple, the algorithm shows good performance in practice with reducing repetitions in long generation, while preserving performance in tasks including reasoning tasks in Math, coding, reasoning etc. The paper provides experiments for Qwen2.5, Llama-3-70B and Gpt-oss-120b, for several benchmarks like MT-Bench, MMLU, GSM8k etc. The algorithm is being used in popular openweight local model repositories like llama.cpp.
Critical strengths
- The paper provides a simple yet impactful LLM text decoding algorithm to avoid repetitions by using a soft penalty for recurring phrases. The algorithm does not modify the decoding if there are no repetitions of length >= L, with L being a hyperparameter. Hence, it preserves the natural decoding performance if there are no repetitions.
- The paper provides several experiments for repetitions and performance on Qwen2.5, Llama-3-70B and Gpt-oss-120b, for several benchmarks like MT-Bench, MMLU, GSM8k etc.
- The paper is also added as a decoding algorithm option in the popular llama.cpp github repository.
Critical weaknesses
- There needs to be more study on the required additional compute for this decoding algorithm. How does the compute require scale with the hyperparam L, do we need to store all seen tokens of length L?
- Although Section I.1 provides ablation on the need of a sequence breaker, it is not completely clear why removing the sequence breaker leads to high false positives of penalized tokens. Further analysis with examples with different choices of sequence breakers would help explain the false positives better.
Questions for authors
Please answer the questions asked in the weaknesses.