- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies uncertainty quantification for ranks: rather than predicting a single rank, it outputs a set of possible ranks. The authors propose Distribution-informed Conformal Ranking (DCR), which transforms the reference distribution induced by uniform rank interleaving into a mixture of potential calibration scores. MDCR is a computationally accelerated variant. Theoretical results support the validity of the method. Simulations and real data analysis demonstrate its advantages.
Critical strengths
- The rank-uncertainty problem is clearly formulated, the methodology is well-motivated, and the proposed approach reduces conservatism relative to the prior TCPR method (Fermanian et al., 2025).
- The paper presents a complete and convincing chain of evidence, including theory, simulations, and real data analysis.
Critical weaknesses
Please see "Questions For Authors".
Questions for authors
- This paper proposes a conformal method for rank uncertainty. The related-work section omits several relevant lines of research, e.g., conformal inference with weakly labeled data (Cauchois et al., 2024) and the statistical literature on asymptotic inference for ranks (Mogstad et al., 2024).
- The Mean-rank baseline appears to perform quite well. Why is that, and what is the actual gain delivered by the proposed method?
- Different ranking models can be fitted on the training data. How does the quality of the ranking model affect the results?
- For the test samples (e.g., the 300 ESOL test molecules), would the method or its results differ if the samples were evaluated one at a time, all together, or in batches?
- For a given sample, if the paper outputs a set of possible ranks, say {4, 5, 6}, do these three ranks have equal status? Could the method provide additional information, e.g., that 4 is most likely, 5 is next, and 6 is least likely?
References
Fermanian, J.B., Humbert, P. and Blanchard, G., 2025. Transductive conformal inference for full ranking. Advances in Neural Information Processing Systems.
Cauchois, M., Gupta, S., Ali, A. and Duchi, J.C., 2024. Predictive inference with weak supervision. Journal of Machine Learning Research.
Mogstad, M., Romano, J.P., Shaikh, A.M. and Wilhelm, D., 2024. Inference for ranks with applications to mobility across neighbourhoods and academic achievement across countries. Review of Economic Studies.