- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Screen-Then-Select is a framework for adaptive temperature selection in LLM mathematical reasoning. It first screens questions using correctness counts from sampled answers, then trains a lightweight MLP to predict a sampling temperature from prompt features. The method is evaluated on MATH, GSM8K, and OlympiadBench, with reported improvements on selected subsets.
Critical strengths
- The paper provides an empirical analysis of temperature sensitivity across questions, including high-sampling experiments and statistical controls.
Critical weaknesses
-
Input-dependent temperature selection and adaptive decoding have been extensively studied in prior work. The proposed framework mainly introduces a correctness-based screening heuristic, without establishing a sufficiently technical contribution in the current version. On GSM8K, the learned selector assigns the same temperature (T=0.9) to every screened question. The reported improvement therefore comes from changing a fixed temperature rather than question-specific adaptation. Furthermore, the evaluation relies on a single rollout block without repeated training or generation runs (test-time scaling by BoN), I therefore have concerns about robustness.
-
The screening procedure requires ground-truth answers and retains questions for which some sampled responses are already known to be correct. If correct answers can already be found, the practical motivation for generating additional answers with a learned temperature is unclear to me. The evaluation only covers small, selected subsets of the benchmarks, without showing end-to-end benefits under a matched inference budget.
Questions for authors
-
Can the authors demonstrate meaningful improvements over a fixed temperature calibrated specifically for the screened population, with repeated runs to establish robustness? By the way, if only math benchmarks are included, the authors can write "LLM mathematical reasoning" instead of "LLM reasoning."
-
What practical inference setting justifies correctness-assisted screening when verified correct answers are already available? How does the method compare with directly using those answers or allocating the same inference budget to additional sampling?
Dear authors,
Please provide a rebuttal by the deadline if you want me to reaccess the paper.