- Rating
- 2
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper focuses on the task of counterfactual fragility certificates. Specifically, for a given fixed setting, it first bulids stress-test sets, and then make the evaluation. The advantage is that this is reproducible. I find this paper hard to read. Specifically, (1) the main contribution of this paper is still not clear to me. It seems that the authors propose a standard procedure (as Fig1). However, I did not find significant contribution here. (2) there is a held-out evaluation, which means the authors evaluate the dataset in some patterns, and then use some held-out evaluation to re-evaluate. I understand this is a OOD ability. However, different patterns may perform pretty differently. I cannot see its significance. Overall, I do not think this paper meet the standard of ICLR.
Critical strengths
This paper focuses on an important question: high-confidence tabular prediction set. The authors propose a procedure that returns verifiable report. That is to say, one can guarantee an inspectable finite audit. I am not an expert in this field, but I do think this is an important task.
Critical weaknesses
Firstly, the writing is somehow confusing. I do think a polish would contribute much to this paper. For example, page 1, the second paragraph, "we address these questions with a fixed classifier and field-level stress tests". I do not understand why this could answer the second question "would checking the record imporve the decision?". This question is answered very late. Overall, there are many confuses in this paper which stops me from understanding the whole paper.
There are a few questions:
- I still cannot understand how this framework distinguish from existing ones. Could the authors provide more evidence on the contributions of this paper, technically?
- It seems that the key point of the method is verifiable. I am not sure why other existing reports are not verifiable. In my view, as long as all the operations are fixed and reported honestly, the report itself is verifiable. So could the authors distinguish how this method is more verifiable?
- The held-out evaluation. I am not sure this is an useful metric. See above.
Questions for authors
See above.