- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
This paper considers the problem of calibration audits which attempt to test whether the predictions made by some system are calibrated. The authors show that standard calibration auditing methods can themselves be biased on some examples even though these examples are actually calibrated, and then introduce PATHTAIL, a calibration auditing method that does not have these same issues.
This submission is very far from my area of expertise, so I would assign very little weight to my assessment of its technical contributions.
Critical strengths
None
Critical weaknesses
The only thing I can reasonably comment on is the writing which, as someone not from this field, I found quite technical even from the first page (e.g., as a non-expert reader of the sentence "In a four-state example, ordinary least squares fitted to eight paths produces a residual of mean −.01814 although the population component is zero.", I do not know what I am supposed to take from this sentence: does the example have 4 hidden states in some Markovian process? why is ordinary least squares relevant here? what is a residual mean? is -0.1814 supposed to represent a large number or is the point only that it is nonzero? finite sample sizes often diverge from their population means, so is it surprising that that happens here as well?). Perhaps these are trivial for people who are more well-versed in this field and it is just too far from my area of expertise, but I could not make ends of the paper.
Questions for authors
None