- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tackles an important problem that is unfortunately common across ML papers, the mismatch between what's reported in the paper and what's statistically valid, as often what's reported is the result of additional hyper parameter optimisation that the comparison baselines do not receive. The paper particularly focuses on the case of sampling procedures, including the common top-k, min p etc.
Critical strengths
I believe it's an important problem and good match for the datasets and benchmarks track, this is a very common problem in ML papers. The proposed methodological solutions/corrections are good and would help identify actual differences in methods.
Critical weaknesses
I mainly just have an issue with the more limited empirical analysis performed for the datasets/benchmark tracks.
Specifically:
- The limited evaluations. Only two datasets and three model families are used.
- Exchangeability vs Expected accuracy: More discussion on these is warranted, and what affect that has on the conclusions.
- The Eight-tier sign tests when the results are not independent
The two datasets limitation is quite important, as in the min-p papers, the authors represent min-p as improving diversity of generated answers, so looking just at accuracy on two math datasets is unlikely to get at the differences or the common use of such samplers, so perhaps understates the differences in this regard.
Smaller point, Table 1 is quite poorly presented and would benefit from reformatting.
Questions for authors
Please see the weaknesses for questions. But the key is that expanded comparisons, and considering metrics beyond just expected accuracy is important here. For the benchmarks track, this warrants more datasets, more metrics, and more comparisons.