- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper evaluates the effects of modifying L and H from the L and H hyperparameters used during training.
Critical strengths
None
Critical weaknesses
The evaluation methodology is described poorly and it is not sound. The number of subsampled problems from each benchmark is very few (e.g. 50 problems for GSM8K, which contains total 1319 test problems). One of the experiments use 12 problems only: The paper itself notes "This analysis uses the smallest sample in the paper an we report no dispersion for it; the claims drawn from it should be read accordingly." The results are not statistically significant. The paper also does not identify the significance of the finding. It is not surprising that deviating from the hyperparameters used during training would negatively impact the performance.
Questions for authors
- What are the results when evaluated over the entire problem set of the benchmarks?
- What is the significance and impact of the findings?