- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks how much evidence about the historical candidate pool is needed to certify that one ranking model beats another, when pool membership is only partially known. The main technical result is an exact algorithm for single-query top-K comparisons, based on the observation that only one shared block straddles the cutoff, with a closed form for adjacent swaps. Experiments use stock scorers on reconstructed A-share pools, a constructed reranking task, and MovieLens.
Critical strengths
The formulation is nice. Looking at the difference between the models' selections rather than at each model's value, and the boundary-block observation, are clean ideas, and the math seems to be correct. The experiments are done carefully and also report the unfavourable results.
Critical weaknesses
The paper is hard to read: Notations and discussions in the main body need the appendix constantly.
The payoff is not shown on real comparisons. The exact Block algorithm can't be run on the natural scorer pairs, whose boundary blocks have thousands of unknown bits (Table 15). It is only used in a constructed reranking, and at width 2 the exact answer is basically the closed form of Eq. (6). On natural comparisons the refinements save very little: Cutoff saves about 0.5% of records over Joint on stocks and nothing on MovieLens, and Contrast saves 1.3% over Frontier on stocks.
The central claim, that certifying a comparison needs much less evidence than recovering all selections, is not really measured on real data. "More than half the memberships unresolved" is measured against full reconstruction, not against fixing all selections. Repairing all 23 errors is guaranteed by construction once a certificate is issued (Sec. 7 says so), so it is not really an empirical finding. In the stock study all memberships are actually known, since pools are rebuilt from market-cap ranks, so the missingness and costs are simulated, and there is no simple non-certified baseline.
Questions for authors
-
Can you report the number of records needed to fix all selections on the same trajectories? A large gap vs Joint/Cutoff would make me more positive.
-
At the same budgets, how often would a simple plug-in winner (known memberships plus the snapshot for the rest) be wrong?
-
Is there a natural setting where models differ only by local reranking, so that Block is actually useful?