- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a decision rule for using ensembles of LLM reasoning results. Stated simply, run a model 3 times. If the 3 answers are not equivalent, run other models 4 further times and synthesize an answer from those results. The authors test under-described models on two benchmarks and find modest gains for their method, at substantially increased inference costs.
Critical strengths
- Paper proposes a way to increase the reliability of LLM answers.
Critical weaknesses
- Submitted to optimization track. This paper does not handle optimization in the machine learning sense, it is entirely dedicated to deciding between using 3 LLM calls or 8.
- Decision on number of ensemble calls is arbitrary and unsupported.
- Choice of Anchor answer, to which other answers are compared, seems arbitrary and is not explored.
- Models used are not stated in the main paper and only briefly mentioned in appendix. Since this paper proposes to explore alternatives to test time scaling, it is egregious to not discuss choice of model and capability.
- Excessive dense notation used throughout, which disguises straightforward ideas and lack clarity. For example, the "fixed equivalence relation" means that answers are case and whitespace normalized to check string equality.
- Devotes large amounts of paper space to superfluous proofs and analysis rather than checking experiments that would make the paper more interesting. For example, Section 3 is a long-winded proof of a simple decision tree with 2 branches: to submit the anchor answer or a synthesized one.
- Section 5 examines win rates over two benchmarks rather than exploring other models, thinking time, or inference costs. Actual performance differences are very small.
- Method requires running multiple models multiple times, which is highly inefficient for inference, rather than considering "depth-wise" test time compute. The synthesis answer also may result in extremely long sequences for inference, and repeated model calls mean that KV caching does not help.
Paper also states:
Executable results were checked through task-specific evaluators and indepen- dent replay; finite mathematical claims were verified by exact or exhaustive computation; and formal results were checked in Lean where indicated.
However there are no Lean checks distributed. This is indicative of over-reliance on AI assistance.
Questions for authors
N/A