- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates how covariance between benchmark scores across training runs affects uncertainty. It proposes a method, SNAP, that estimates cross-benchmark covariance across various training runs to separate item-specific variation from covariance that generalizes across questions. It shows, through various experiments, that: (1) the benchmarks included in the estimation can have significant impact, and (2) independence assumptions can change the standard deviations of the average margin scores by 24%. The authors recommend to report standard deviation of benchmark averages across training replicas and repeat the calculation after removing each benchmark.
Critical strengths
None
Critical weaknesses
This paper has limited novelty, weak demonstrated benefit, and its presentation is difficult to follow because many paper-specific terms remain undefined at the moment of their introduction. I list the two critical weaknesses below.
1. Poor writing
By far the weakest point of the paper is its poor writing. The paper contains many undefined terms that are not explained at the time of their introduction and includes many numbers that are very hard to interpret, but still left to the reader to interpret. For instance, these are all undefined terms in the abstract:
- DataDecide/PolyPythias: niche training runs, appears sudden and are unexplained.
- per-byte margin: author-defined term that is only explained in section 2, but carries no meaning for a reader at this point.
- margin comparison that assumes independence: not explained what kind of independence is assumed, or what a margin comparison is
- 0.11: not explained what practical meaning this value has
- alone in its format: not explained what it means for BoolQ to be "alone in its format", a reader has to wait until L273 where it is mentioned in brackets that it refers to the yes/no nature of this benchmark.
- cross-half estimator: cross-half has no meaning at this point. Also, from the text alone it is unclear that the authors propose SNAP.
- net of item noise: not sure what it means to be net of item noise
- fails at 1.217: not specified why the arbitrary looking number 1.217 indicates that this experiment failed.
- gives an inflation of 1.237 here: what is "here", what does it mean to have an inflation of 1.237?
These writing errors and mistakes appear everywhere in the paper. Additionally, the goal of the paper is insufficiently motivated and explained.
2. Incremental Nature
The authors do not convincingly show the practical benefit of their problem and solution: it remains unclear who would use the method and why. Overall, the paper is also very incremental: no ground-breaking experiments or interesting observations are done. While subjective, at no point while reading the paper I felt surprised or interested in the conclusions being presented.
Overall, the proposed method does not address a real, motivated problem. The authors should clarify:
(1) Who would use their method and who the recommendation is aimed at
(2) Find practical and interesting scenarios where ignoring cross-covariance between benchmarks would lead to wrong conclusions
(3) What value their work adds in those cases, and show explicitly that their method would avoid those wrong conclusions
Additionally, the authors themselves indicate several times that most analyses were exploratory and at least sometimes inconclusive, further decreasing the value of the work.
Questions for authors
See above