- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors tackle the problem of distinguishing answerable and unanswerable questions. For an answerable question, there exists a correct answer, while an unanswerable question is controversial, ambiguous, based on a false assumption, counterfactual, unsolved, or about a future unknown. They find that existing uncertainty estimates (verbalization, self-consistency, ensemble) and a fixed combination do not do well. They propose DACR, which first predicts whether the question is answerable or unanswerable, then uses this prediction to decide how to combine the uncertainty estimates. DACR leads to better results.
Critical strengths
- They find that the answerability of a question is much more easily predicted from the question itself than from models’ behavior (i.e. uncertainty estimates based on verbalization, self-consistency, ensemble). This is a mild contribution, since it means that we can detect unanswerable questions with minimal means.
Critical weaknesses
- The problem tackled in this paper – distinguishing answerable and unanswerable questions – has very little practical utility and thus severely limits the impact of the work. The typical and useful usage of confidence/uncertainty is to estimate the probability that a given prediction is correct (e.g. to abstain when the probability is low). The key difference from the problem tackled in this paper is that even if a question is answerable (i.e. there exists a correct answer), the model may not know the answer, which means that it is best to abstain. Thus, the basic premise that answerable questions should always be answered is flawed.
Questions for authors
- How is the within-slice AUC defined, e.g. in Figure 2? My guess: for the answerable class, the AUC is computed on questions that are answerable or unanswerable (any type t); for each unanswerable type t, the AUC is computed on questions that are answerable or unanswerable type t.
Thank you for the clarifications. I will maintain my score, given that the authors' response confirms my W1.