- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes evidence contracts to distinguish information used for prediction from evidence that can support a semantic claim. It uses four-state observations to construct admissible concept bounds and provides conditional guarantees under reporting shift. The experiments include synthetic reporting-shift settings, text diagnostics, and real questionnaire data.
Critical strengths
I think the distinction between predictive information and admissible evidence is worth studying. The paper also provides theoretical analysis and several experimental comparisons. I appreciate that the authors explicitly check their assumptions and acknowledge where the guarantees do not hold for the learned models.
Critical weaknesses
-
My main concern is the gap between the theoretical guarantee and the actual experiments. Proposition 4 requires a pointwise condition, but Section 5.2 shows that all five Round-3 learned heads violate it on 103–141 out of 4,096 concept configurations at α = 0.1. Eq. (7) is a cross-entropy training objective and does not enforce this condition. I understand that the authors acknowledge this limitation, but I still don't see how the nominal probability guarantee can be applied to the learned models. The reported selective errors are empirical results, and the slack-based bound is considerably weaker.
-
I am also not convinced by the novelty. The Introduction contains no citations, and although Section 2 discusses related work, I still find it difficult to identify the main technical advance over existing concept-based models, monotone prediction, and selective classification. The paper also builds on the original dual-map system, but I don't think the difference is explained clearly enough.
-
The presentation could be clearer. Figure 1 does not explicitly show the semantic head or the certification step. It also makes the envelope look like an output of completion, whereas Eq. (5) computes its endpoints directly using the semantic head. I found this confusing when trying to understand how the different parts of the method fit together.
Questions for authors
-
How can the pointwise assumption in Proposition 4 be enforced or verified for learned heads? Without this, what guarantee can actually be provided in practice?
-
What is the main technical contribution beyond existing methods and the original dual-map system? I would appreciate a clearer explanation of this distinction.
-
Could the authors clarify the relationship between completion, the semantic head, and the output envelope, particularly in Figure 1?