- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets at the demographic-unfair false alert problem introduced by one-to-many name screening, by adapting target-decoy competition from proteomics, i.e., comparing scores of the real list and the generated decoy lists. The idea is smart and straightforward. The paper also provide some theoretical guarantees. However, my biggest concern is that the proposed decoy constructions do not achieve the exchangeability assumption required by the theoretical results.
Critical strengths
- This paper provide a clear problem formulation to the name screening and unfair false alert problem. The authors also provide formal theoretical results under an explicit exchangeability assumption, covering composition amplification, performance guarantees, misspecified decoy consequences, etc.
- The experiments evaluate several matcher families, multiple decoy sources and generation strategies, matched alert volumes, detection recall, and multiple matcher families, providing a useful view of the trade-offs involved.
Critical weaknesses
CW1. The theoretical guarantees rely on the exchangeability assumption that practical decoys are hard to satisfy. The empirical results show substantial deviations from nominal false-alert levels for several practical decoy constructions, roughly between 2.3α and 11.3α. Thus, the formal guarantees do not directly apply to the proposed decoy constructions. CW2. The per-country threshold baselines simply assign a threshold for each country. They are not sufficiently strong for evaluating the value of demographic-blind calibration. Stronger country-aware baselines shoule be evaluated. CW3. The demographic-blind claim needs to be further clarified. During decoy constructions, using the country information helps to meet the exchangeability assumption but violates the demographic-blind cliam; not using the country information remain demographic-blind but can lead to severe violation of the exchangeability assumption.
Questions for authors
- Can the authors provide stronger evidence that the proposed decoy generator/strategy satisfies the exchangeability assumption?
- How does TDC compare to stronger country-aware baselines?
- Can the authors clarify CW3?