- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes "Early Release" to improve stratified arithmetic sampling for LLM. It switches to normal sampling once the generated prefix is fully inside its assigned stratum. The authors provide theoretical guarantees and show improved answer diversity and coverage across different models and tasks.
Critical strengths
- Simple and effective method. I like this type of work that makes a small but clear improvement. The problem is well motivated, and the proposed solution is simple. It does not require any model training or modification.
- Good theoretical support. The authors prove that "Early Release" preserves the joint distribution of stratified arithmetic sampling under exact arithmetic. The proof gives a clear reason why the method works, rather than only showing empirical improvements.
- Comprehensive experiments. The experiments cover many models, sampling settings, and benchmarks. The authors also provide ablation studies, runtime comparisons, and statistical analysis. I appreciate that they report negative results on reasoning and coding tasks.
- Clear writing. Overall, the paper is well written and easy to follow. The motivation, method, and results are presented clearly.
Critical weaknesses
- Gap in model sizes. There is a large gap between 7B and 70B in the experiments. Adding 14B or 32B models would make the results more complete.
- Numerical precision. The proof assumes exact arithmetic, but the implementation uses float64 to update I(h). It is unclear whether rounding errors near stratum boundaries could cause incorrect early release.
- Lack of analysis on where diversity appears. The method shows limited gains on math, code, and reasoning tasks. It would be useful to analyze when meaningful diversity appears during generation, and how this relates to the release position and the improvement.
- Limited benefit over simpler methods. Figure 6 shows that coordinating only the first token already gives most of the improvement. The additional practical benefit of the full method is not very clear.
- Runtime overhead. Early Release is still around 12% slower than independent sampling on average. A comparison under the same inference time budget would be useful.
- Figure quality. The figures could be improved, especially Figure 1. Some elements are small, and the main idea could be shown more clearly.
Questions for authors
Please see weaknesses section, and also some additional questions here:
- Follow up on weakness #3: Is there a relationship between the position of semantic differences, the release position, and the final improvement?
- Follow up on weakness #4: When is coordinating more tokens actually useful?
- How sensitive are the results to token ordering in arithmetic sampling? Would different orderings change the diversity improvement?
- Could the authors discuss when Early Release should be preferred over Lattice Sampling in practice?
Thanks for the rebuttal and additional experiments. Most of my concerns are addressed. The remaining limitations primarily lie in the relatively modest absolute performance gains for certain tasks, as well as the relatively high computational overhead for short outputs and small models. I therefore maintain my current rating.