- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces semantic entropy for the cascading decision: run a small model several times, use semantic clustering to determine its consistency, and threshold that to see whether a bigger model needs to be run on the task. The authors evaluate their approach on several benchmarks, finding that it sometimes does not outperform a difficulty-decision baseline, frequently spends too much queries on the small model to determine consistency, but can sometimes slightly outperform the baseline.
Critical strengths
None
Critical weaknesses
The experiments presented in this paper are outdated and do not show a clear gain for the method. In particular:
- The related work section is not sufficient to thoroughly describe a field that has a lot of work taking place in it. The paper also shows negligence when citing other works. For instance, Mallen et al. only occurs as a citation after their benchmark "PopQA" has already been featured prominently and the paper behind GSM8k is not cited (despite featuring even more prominently).
- Several included subsections carry no value. For instance, section 4.7 only talks about why the benchmark did not prove suitable for the project, which has no additional value.
- Writing, in particular in the experimental section, is very poor. As an example, L147-149 does not even write full English sentences.
- The authors propose a method that is extremely similar to already proposed methods, with the only difference that they requery the same model instead of querying different models. Thus, at most the value of the work would be very incremental.
- The experimental section is unclear, with no experiment providing a thorough and interesting analysis. Results are described unclearly and do not give an indication of why the method is useful.
- The authors claim that the method itself is not a contribution, but the evaluation methodology. However, the experiments are poorly written, and do not provide value for the community.
- The paper seems almost entirely LLM-written. Not a problem by itself, but it seems that no attempt was made to improve upon its output in any significant way.
Questions for authors
None