Split Conformal Prediction with Tunable Prediction Set Size

Conformal Prediction (CP) is a model-agnostic framework for constructing prediction sets with finite-sample coverage guarantees and has become a key tool for reliable decision-making in high-risk applications. However, a limitation of CP is that the size of the prediction set is implicitly determined by the target coverage level, which can result in sets that are too large to be actionable. This is particularly problematic in applications such as medical diagnosis, where there are many possible conditions and decisions must be made under time constraints. To address this limitation, recent work studies CP procedures in the Tunable Prediction Set Size (TPSS) setting, wherein the user specifies a maximum allowable prediction set size, and the goal is to provide valid coverage guarantees under this constraint. While Split CP (sCP) is the most widely used variant of CP, enforcing such size control within this framework is challenging due to its quantile-based calibration procedure. As a result, existing approaches for TPSS rely on the more flexible Conformal e-Prediction (eCP) framework, which enables data-dependent size control but inherits limitations such as weaker structural properties and less accurate coverage approximation. Motivated by this gap, we develop Tunable-sCP (TsCP), a method to address the TPSS problem within the sCP framework, enabling explicit control over prediction set size while preserving the key structural properties of sCP. We further show that our leave-one-out construction yields a substantially more accurate coverage estimate than eCP for the same calibration size, reducing conservativeness and making it more reliable in practice. Empirically, we demonstrate on synthetic and real-world datasets that TsCP consistently improves the empirical coverage–efficiency trade-off compared to prior approaches.