- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how to select the model when the test data are from heterogeneous groups and the group membership is missing. The paper first presents a negative result that missing linkage can result in non-identification (Theorem 1) and then proposes an algorithm to use available and selectively disclosed links to verify that a fixed candidate is within a prescribed tolerance of the best candidate. A numerical study evaluates the proposed method.
Critical strengths
-
Interesting problem and useful framework. The problem itself is interesting and can have some useful applications. The paper's idea is to move from recovering a complete ranking to certifying an acceptable choice. This distinction is valuable and makes the problem more practical.
-
Clear theoretical results. The first negative results on ranking ambiguity are clear and motivate the later focus on certifying an acceptable choice. Section 4 on selective linkage provides concrete procedures for certifying an acceptable model choice and clearly quantifying the disclosure required.
Critical weaknesses
-
The presentation is difficult to follow. Dense notation, terminology, and frequent transitions between ranking ambiguity, fixed-rule certification, minimum proof size, and online discovery make the paper extremely difficult to understand. A consistent example and clearer separation of these concepts would make the paper easy to understand.
-
The practical usefulness of the proposed method remains unclear. Although certificate verification is efficient, discovering the required links can be costly. Thus, the theoretical certification results do not provide a convincing practical method.
Questions for authors
-
Can the authors provide a simple illustrative example that explains the setting and the certification procedure without extensive notation? Can the authors also clearly express the logic behind the theoretical contribution?
-
How are informative links selected? If the missing linkage is due to privacy issues, how is information leakage assessed when additional group memberships are revealed? If the motivation is the cost of linkage, can the authors provide some examples to motivate the problem?
-
How practical is the method in a realistic application, considering the cost of acquiring and authenticating links and the computational burden of link selection? Could the numerical study more clearly evaluate different tolerances, comparing the proposed method against a fully informed oracle and simple baselines?