- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors study what they call the "Arithmetic Error Bar", which to my understanding is supposed to measure a degree of variance that is usually ignored in reporting accuracy on benchmarks. They argue that simple accuracy scores obscure potential flipped answers, and that their approach is a means to account for this variation. That running the same model with greedy decoding on different computational architectures has variation is an important fact, and it is interesting to note that some of the uncertainty in language models are likely due to the hardware, and not the model per se. They study three small language models and find that their measurement holds for 2 out of 3 models. They claim that the mechanism of this numerical variation is determined by a discrete threshold.
Critical strengths
The object of study is of interest with respect to the reliability of models. AI systems are often discussed without taking the underlying hardware into account, and the paper thus points to an interesting question. However, it could be clearer that this is the actual focus of the paper. I find the paper very hard to parse, and therefore struggle to single out the actual contribution and potential implications of the findings. This could potentially be alleviated by a general re-writing of the paper and general clarifications that clearly explain the methodology and implications / contributions of the work.
Critical weaknesses
- The paper is generally very hard to parse.
- It is not exactly clear what the contribution is, and as a non-expert reader there is no background on the theory of which the work is based.
- While this is also mentioned in the limitations the approach is tested on three models wherein it applies to two of them (that are only tested on CPUs).
Questions for authors
- Can you clarify the key implications of your work?
- How would this apply if models were run on GPUs? Almost all inference is done on GPUs, which begs the question of whether the finding also holds there.
- What questions are the models being evaluated on, and how is the evaluation performed?