- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper measures the gap of reliability that can arise under covariate shift on a level-confidence setting. The paper defines "fragility profile", which identifies the propensity distribution. Under low levels of confidence-preserving covariate shifts, variance itself is sufficient, while in high-level shifts it is insufficient. This is derived theoretically (Sec.3) and evaluated experimentally (Sec 5. What the full~). To make the framework applicable in practice, they propose a confidence-binned and readout approaches.(Sec.4) Experiments on ImageNet classifiers evaluate the estimated frgility profiles, compare them with variance-based methods, and examine relevance and limitations.
Critical strengths
- The introduction of the fragility provides a fresh perspective. This characterization on analyzin reliability is an interesting approach.
- The paper provides extensive experimental evaluations to support its various theoretical and methodological claims.
Critical weaknesses
-
This paper has MAJOR writing issues. The explanation of the main idea is not only insufficient, but leaves the core components and implications highly ambiguous, and jargons used without proper definition. Upon reviewing, I had to go over the written statement repeatedly to at least guess what the authors were trying to argue. The sentences are overly abbreviated, and almost no intuitive explanation or motivation for the possible design choices. To list a few, (Line number),
- (47) What is budget? Why is it this treated as a “budget”? Why can the \rho value be seen as a budget? Why is worst-case reliability drift a function of a budget that is \chi-square supremum difference?
- (53) Why is stop-loss transform applied?
- (48-50) What is P? What is \rho_+ ? why is this interval used, and why is it not defined?
- (124) What is Q?
- (235) What is readout? what is finite readout? what is nested readout? what is learned readout?
- What is R_1 and R_2? What is \sigma? Is it \sigma-algebra or standard deviation?
- What is cell? Why is cell needed needed? What is readout cell? What is \pi?
- (244) What is binned "oracle”?
- (248) : “The proof is conditional Jensen and nested feasible sets.” Is this a logical English sentence?
- (264) What is "outer”? Is there "inner”?
- (264) What value does the method "plug-in”? These questions arise while reading through the paper, but no answers can be found, I only had to guess through.
-
Despite the extensive experimental evaluation, the presentation lacks a clear focus on the paper's main contributions. Numerous experimental results are presented with similar emphasis, making it difficult to distinguish the key findings from supplementary analyses. Most of the experiments including tables and figures are placed in appendix, which hinders the reader to grasp the essential concepts. Only an abbreviated version is contained in the results section, most of the tables and figures are placed outside the main paper.
Questions for authors
- Could the authors clarify which theoretical and empircal findings they consider the central contributions of the paper? Also, could authors clarify how the main experiments support the contributions?