- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
From authors: "Survival under scalar recalibration is insufficient evidence of contextual value beyond a simpler predictor with the same side information." "The contribution is a documented empirical failure of an attribution criterion, rather than a new calibration algorithm."
The authors argue that calibration depends on the information available to the recalibration step (e.g., the context, the order, etc.). They should this impact with examples in two real-world datasets.
Critical strengths
- The authors explore calibration and the impact of certain system characteristics on its calibration, which is a relevant topic.
- The authors test it on real datasets where such a challenge might exist.
- The authors apply some data augmentations (i.e., transformations, sampling).
Critical weaknesses
-
The main claims, goals, and guarantees are unclear. The paper reads more as a high-level suggestion to be careful about the context used in calibration, based on empirical results from two datasets. It offers little substance on what to do, how to measure the effect, or how to identify it, how to improve it.
-
In particular, the conclusion that calibration depends on the information given seems somewhat obvious.
Questions for authors
- Have the authors considered using synthetic datasets to evaluate this challenge more broadly?
- Isn't the canonical swap a type of data augmentation? Is this already used by literature? What is the difference?
- Regarding the weakness above: the conclusion that calibration depends on the information given seems somewhat obvious. Could you explain, concisely and in plain English, what I might be missing here?