acceptodds
Under review as a conference paper at ICLR 2027

Conformal Confidence Suppression for Offline and Online Learning-to-Rank

Abstract

Ranking systems often decide which candidates remain eligible before assigning display positions. Suppressing a useful candidate at this stage can prevent its recovery by the downstream ranker. We propose conformal confidence suppression (CoCo), a model-agnostic gate that suppresses only when a calibrated upper bound on incremental utility is negative. We distinguish eligibility from actual display, construct global, groupwise, and locally scaled gates, and prove finite-sample false-suppression control for an exchangeable calibration target. We also develop gated policy-value estimators and an online predict–gate–rank–update procedure, with conditional calibration results and a suppression-aware regret decomposition. On a public learning-to-rank benchmark, CoCo-M improves utility over the same ranker while empirical target violation tracks the chosen risk level. A sequential full-information simulation shows gains for evolving rankers, and randomized uplift data provide complementary calibration evidence. The results characterize how calibration and gate granularity trade off utility and conservative suppression.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal prediction, learning to rank, uncertainty quantification, online learning, counterfactual learning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_w7htOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This submission proposes Conformal Confidence Suppression (CoCo), a model-agnostic approach to controlling candidate eligibility in LTR systems. The method distinguishes eligibility from actual display and estimates the incremental utility of keeping a candidate eligible, rather than suppressing it. Instead of suppressing candidates based on point estimates of negative utility, CoCo uses conformal upper prediction bounds to make conservative suppression decisions.

The theoretical analysis provides finite-sample false-suppression guarantees under exchangeability assumptions, along with results concerning offline policy evaluation and online regret decomposition. Experiments on MSLR-WEB10K, synthetic online ranking, Criteo uplift data, and Open Bandit logs examine the tradeoff between suppression, ranking utility, and calibration

Critical strengths

A relatively comprehensive mathematical formulation. The paper connects conformal prediction, causal uplift estimation, and counterfactual ranking within a unified framework. It explicitly distinguishes the calibration target from conditional expected uplift and acknowledges the assumptions required for its guarantees.

Critical weaknesses
  1. Limited methodological novelty and insufficient motivation. The core method is essentially an application of standard one-sided conformal prediction. The theoretical results largely follow from standard exchangeability arguments, off-policy estimation properties, and elementary risk bounds. Also, I am not convinced that the proposed suppression problem is sufficiently motivated in practice, as modern ranking systems typically optimize objectives that incorporate user and business utility. It is unclear which practically important limitation of existing utility-aware ranking systems the proposed method addresses.

  2. Experimental evidence does not convincingly establish practical benefits, and reproducibility looks limited. Empirical results are heavily artificial or simplified. No executable code or supplementary implementation is provided which is expected given the relatively modest computational scale and the dependence of the results on implementation details.

Questions for authors

Can the authors provide a concrete practical setting where a utility-aware ranker with an explicit no-display option cannot adequately handle candidate suppression? What specific limitation of existing ranking objectives or serving architectures makes a separate conformal gate necessary?

How should the display-cost parameter used to construct net utility on MSLR-WEB10K be interpreted in a real ranking application?

Reviewer_GWZTOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work proposes a conformal prediction based gate method to suppress ineligible candidates before sending them to the actual ranking system. In order to prevent false suppression, the authors propose three conformal gates, CoCo-1, CoCo-M, and CoCo-F, and provide theoretical false suppression control guarantees. The authors present COCO implementations in both online and offline settings, and demonstrate the effectiveness of COCO by comparing rankers equipped with CoCo gate againt ungated rankers and rankers with vanilla point gate in experiments.

Critical strengths

The problem is motivated nicely. The problem formulation and notations therein are described clearly.

Critical weaknesses

1, The presentation of the paper needs to be improved. For example, in Section 4.1, the roles of each paragraph and the connections between them are not clear to me, as no narrative is given to explain how the concepts are organized. Additionally, what is the definition of μ^1,β^1\hat{\mu}_{1,\hat{\beta}_1} in Line 162, as β\beta is not introduced previously?

2, Although the authors demonstrate COCO can get higher utility over baselines through experiments, they do not provide theoretical utility (or regret) guarantee for COCO except for Proposition 4, which has only linear regret bound. This is critical as I think theory part should have been the major contribution of this work.

Questions for authors

1, What is NDCG in the Experiments section, and how is it related to the provided theory?

2, Have multiple base rankers been tried? I wonder if a better (or worse) ranker will make the comparison results different.

3, How should we know which variant of COCO to use beforehand?

Reviewer_A6XXOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies a suppression gate for learning-to-rank: before a ranker assigns positions, a gate h(z)∈{0,1}h(z)\in\{0,1\} decides whether each candidate remains eligible for display. The proposed method, CoCo, combines uplift modeling (a T-learner or a DR pseudo-outcome regression) with one-sided split conformal calibration: a candidate is suppressed only when the calibrated upper bound Uα(z)=Δ^(z)−qαU_\alpha(z)=\hat\Delta(z)-q_\alpha on a calibration target Γ\Gamma is negative. Three variants (global, groupwise, and locally scaled calibration) are given, together with finite-sample coverage statements. The novelty of the method itself is limited, but the design is reasonable.

The paper describes both offline and online LTR settings (Sections 4.2–4.3) and evaluates many variants. However, the LTR experiments (MSLR-WEB10K and a synthetic online stream) are full-information evaluations of the gate with a fixed ranker; the logged-data ranker objective of Section 4.2 and the bandit-feedback setting of Section 4.3 are not tested (acknowledged in Sections 4.2 and 6.2). Criteo and Open Bandit have no top-K stage (Appendix C.4, C.6).

My main concern is that the problem formulation and the experiments are not aligned. Section 3.3 states that "The objective is to achieve high utility while controlling suppression risk," but the experiments never report the suppression risk RFSR_{FS} defined in that section, for any gate, in any table.

Critical strengths
  1. The method is simple and easy to understand. The explicit separation of gate-level eligibility HH from realized display AA (Sections 3.1–3.2, Appendix C.2), and the observation that non-display caused by ranking below the available slots is not a suppression-side example, is practically relevant and well argued.
  2. The experiments use several public datasets (Criteo Uplift v2.1, MSLR-WEB10K, a synthetic online stream, Open Bandit) and include the ungated ranker and a point gate as baselines in every table. As discussed below, however, the reported quantities do not match the stated objective, so the comparisons cannot currently be interpreted either way.
Critical weaknesses

1. The stated objective is not evaluated, and the experiments as designed cannot show the claimed benefit

(a) The stated objective is not evaluated: false-suppression risk is never reported

Section 3.3 defines the false-suppression (FS) risk RFS(π)=P{h(z)=0, Δ(z)≥0}R_{FS}(\pi)=P\{h(z)=0,\ \Delta(z)\ge0\} and the harmful-impression (HI) risk RHI(π)=P{h(z)=1, Δ(z)<0}R_{HI}(\pi)=P\{h(z)=1,\ \Delta(z)<0\}, and states the objective as high utility while controlling the FS risk. The experiments instead report "violation" Pr⁡n(Γ>Uα)\Pr_n(\Gamma>U_\alpha) (Table 1 caption; Appendix C.1), and only for conformal gates. This column equals ≈α\approx\alpha by construction (it is the empirical coverage of the conformal bound) and therefore carries no information about the gate's performance: it is 9.8% both for CoCo-1, which suppresses nothing, and for CoCo-M, which suppresses 39.4% (Table 7). The quantity the formulation asks for, Pr⁡n(Uα<0, Γ≥0)\Pr_n(U_\alpha<0,\ \Gamma\ge0) (or the same event with Δ\Delta in place of Γ\Gamma), does not appear in any table.

This omission cannot be attributed to identifiability. On MSLR the target is full-information net utility, so Γ=Δ\Gamma=\Delta is known for every test document. In the online simulation the latent Δ\Delta is known, and the paper already computes Pr⁡n(Δ>Uα)\Pr_n(\Delta>U_\alpha) (Table 2) while noting in Appendix C.7 that {Uα<0,Δ≥0}⊆{Δ>Uα}\{U_\alpha<0,\Delta\ge0\}\subseteq\{\Delta>U_\alpha\}. In both cases the FS rate can be computed from quantities the paper has already computed. The version with Γ\Gamma, Pr⁡n(Uα<0, Γ≥0)\Pr_n(U_\alpha<0,\ \Gamma\ge0), is computable wherever violation is, including Criteo and Open Bandit.

The same applies to the point gate. Section 6.1 states that "Point's false-suppression risk was not recorded, precluding a matched-risk comparison." This is precisely the comparison the paper's motivation calls for. From the paper's own numbers one cannot rule out that Point already satisfies the risk budget. For any gate, as fractions of all test documents, FS−HI=suppression rate−fraction of negative candidates,\mathrm{FS}-\mathrm{HI}=\text{suppression rate}-\text{fraction of negative candidates}, which for Point at cost 0.10 is 89.7%−83.9%=5.8%89.7\%-83.9\%=5.8\% (Tables 7 and 5; this assumes the suppression rates in Table 7 are over all test documents, which the paper does not state). If Point's HI rate is small — plausible for a regression fit to a target that is 83.9% negative — then Point's FS rate is within α=0.10\alpha=0.10, in which case CoCo-M is a gate that satisfies the same risk constraint but has lower utility (0.2848 vs 0.3104, Table 1). The paper should report the FS and HI rates for all gates including Ranker and Point, and compare utility at matched FS.

(b) The motivation is not encoded in the evaluation metric, so the experiments cannot show the claimed benefit

Section 1 motivates conservative suppression by the irreversibility of suppression: a suppressed candidate cannot be recovered by the downstream ranker, whereas an eligible candidate is still subject to ranking. This is an argument that false suppression is more costly than harmful impression. The utility V(π)V(\pi) of Section 3.3, however, counts both error types in the same units. Under a symmetric utility, and up to interactions between candidates, the gate that maximizes expected utility is 1{Δ(z)≥0}\mathbb{1}\{\Delta(z)\ge0\}; with an estimate Δ^\hat\Delta that is unbiased for Δ\Delta, this is the point gate, and a one-sided margin −qα-q_\alpha can improve utility only insofar as it corrects bias in Δ^\hat\Delta. This is consistent with Table 8 (CoCo-M utility rises monotonically toward Point as α\alpha grows) and Table 11 (CoCo-M exceeds Point only at α=0.40\alpha=0.40, where the margin is essentially zero and the remaining effect is per-group bias correction). As designed, the experiments therefore cannot demonstrate the benefit the motivation claims, and the observed ordering Point >> CoCo in utility is the expected outcome rather than a surprising negative result. Either an asymmetric cost for false suppression should enter the objective, or the comparison should be made at matched FS (see (a) above).

(c) Missing obvious baselines

  • A tuned-threshold point gate 1{Δ^(z)≥τ}\mathbb{1}\{\hat\Delta(z)\ge\tau\} with τ\tau chosen on the calibration split to match a target FS (or suppression rate). CoCo-1 has exactly this form with τ=qα\tau=q_\alpha; without this baseline the contribution of conformal threshold selection cannot be separated from the effect of thresholding at a nonzero value.
  • On MSLR, the oracle gate 1{Γ≥0}\mathbb{1}\{\Gamma\ge0\} with the same ranker. It is computable in a full-information benchmark and gives the utility upper bound of gating; without it the reader cannot tell what fraction of the achievable gain CoCo-M's +0.0295+0.0295 represents.

2. Unspecified design choices and an overstated contribution statement

(a) CoCo-M's group definition is unspecified, and CoCo-1 is degenerate

  • Section 4.1 lists possible group rules for CoCo-M ("query type, user segment, item type, or a score bin fixed before calibration"), but the group G(z)G(z) actually used is stated only for the online simulation ("the three context groups", Appendix C.7). For MSLR and Criteo it is not specified anywhere. The main MSLR result (CoCo-1: 0.0% suppression, CoCo-M: 39.4%, Table 7) therefore depends on an undocumented design choice. If GG is a bin of Δ^\hat\Delta, group-conditional validity has a quite different meaning from the interpretation in terms of query types, user segments, or item types suggested in Section 4.1. The authors should state what GG is and confirm that it was fixed before seeing the calibration data.
  • CoCo-1 at α∈{0.05,0.10}\alpha\in\{0.05,0.10\} suppresses 0.0% of candidates (Tables 7 and 8) and is therefore numerically identical to the ungated ranker in Table 1. The claim that "CoCo-M improves on global CoCo-1 at the conservative budgets" (Figure 1 caption) thus reduces to the statement that suppressing some candidates is better than suppressing none, and does not support groupwise over global calibration. The jump of CoCo-1 from 0.0% to 64.7% suppression between α=0.10\alpha=0.10 and 0.200.20 (Table 8) suggests that the global quantile falls at a discontinuity induced by the discrete net-utility target; this should be discussed.

(b) The contribution statement overstates what is new relative to prior CoCo work

The method name, the three variants CoCo-1/M/F with the same global/group/local structure, and the idea of conformal confidence bounds on uplift predictions for content suppression appear in a published SIGIR 2026 short paper ("CoCo: Conformal Confidence Suppression to Optimize Search Results," DOI 10.1145/3805712.3809847), which is not cited. The second contribution, "we introduce CoCo-1, CoCo-M, and CoCo-F, three conformal gates" (Section 1), is therefore not accurate as stated. The actual difference with respect to that work is real but narrower: an explicit calibration target Γ\Gamma (and the acknowledgement that coverage of Γ\Gamma is not a guarantee for Δ\Delta), groupwise quantiles for CoCo-M (the earlier CoCo-M uses a pooled quantile of group-normalized residuals), a fitted scale model instead of an ensemble for CoCo-F, the DR option, the separation of gate from display, the proofs, and evaluation on public benchmarks. The paper should cite the earlier work and state this difference explicitly; my assessment of novelty above is based on it.

Questions for authors
  1. Please report the FS rate Pr⁡n(U<0, Γ≥0)\Pr_n(U<0,\ \Gamma\ge0) and the HI rate for every row of Tables 1, 2, 4, 7, 8, and 9, including Ranker and Point.
  2. What is G(z)G(z) for CoCo-M on MSLR and Criteo? Was it fixed before calibration?
  3. At matched FS, does any CoCo variant exceed the utility of a tuned-threshold point gate?
  4. Please clarify the relationship to the SIGIR 2026 CoCo paper and state the difference in the contribution list.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all Reviewers

We thank the reviewers for their time and detailed feedback. The reviews identified gaps in how we evaluated the stated objective and described the calibration protocol. We decided to withdraw the submission to address these issues. Before withdrawing, we would like to respond to the main questions.

False suppression, harmful eligibility, and the Point comparison (to Reviewer A6XX). We agree that bound exceedance is not the false-suppression rate. The relevant events are suppressing a candidate with a nonnegative target and keeping a candidate eligible with a negative target. The latter is an eligibility error; it need not result in display. On MSLR and in the online simulation, the benchmark target or latent utility is available, so the omission was an evaluation gap. For example, the DR Point gate’s 89.7% suppression rate and the 83.9% negative-target share on MSLR imply that false suppression exceeds harmful eligibility by about 5.8 percentage points. These aggregates do not determine whether Point meets the 10% budget. We therefore cannot answer from the submitted results whether CoCo achieves higher utility at matched false-suppression risk. We agree that the revision needs direct error rates for every gate, a threshold chosen on held-out data for the same risk budget, and an MSLR oracle comparison. CoCo-1 is itself a global threshold rule. At the two smaller MSLR budgets it suppresses no candidates, so the comparison to it does not by itself establish the benefit of groupwise calibration.

On Criteo, the archived DR results give false-suppression and harmful-eligibility rates of 1.93% and 74.15% for Point, and 1.11% and 78.77% for CoCo-F at α=0.10\alpha=0.10, measured against the DR pseudo-outcome. These are target-level diagnostics. Randomized treatment helps identify conditional mean effects, but the sign of a noisy pseudo-outcome does not identify the sign of an individual or conditional causal effect.

Groups and implementation (to Reviewers A6XX, GWZT). In the archived MSLR runner, CoCo-M bins predicted uplift using up to five quantile groups. In Criteo, it bins predicted control conversion risk, also using up to five groups. In both runs, the cut points were estimated from predictions on the calibration split and then applied to the test split. They were not fixed on a separate split before calibration. The MSLR runner also fits the CoCo-F scale using calibration residuals. We should have documented these choices and distinguished those empirical runs from the split-calibration conditions used in the theory. For deployment, our starting choice would be CoCo-1 for a pooled risk target, CoCo-M when meaningful groups can be specified in advance and have enough calibration data, and CoCo-F when residual scale varies with context and can be fit on separate data. We have not established that one variant is uniformly preferable.

Presentation, rankers, and theory (to Reviewer GWZT). In Section 4.1, β^h\widehat{\beta}_h denotes the fitted parameters of the outcome model for arm H=hH=h. We should define it at first use and explain the sequence from fitting uplift, to constructing the calibration target, to choosing the eligibility rule. NDCG@10 measures discounted relevance in the displayed list relative to the ideal top ten from all candidates for that query. It excludes display cost and is a separate ranking-quality metric. The false-suppression theorem does not imply an NDCG guarantee. Appendix C.5, Table 7, tests relevance-trained and net-utility-trained LambdaRank models. The qualitative utility ordering is similar for both, but they share an architecture. Proposition 4 is a regret decomposition under an additional value-loss assumption. The current analysis does not establish sublinear online regret or a utility advantage from calibration.

Practical role and evaluation scope (to Reviewers A6XX, w7ht). A ranker with a no-display option can implement the same risk constraint if it has the necessary uncertainty estimates and can enforce the constraint. A separate gate is useful when an existing ranker allocates slots by relevance and a different decision determines whether a module should be eligible based on its incremental effect. For example, an eligibility rule can withhold a shopping module when there is enough evidence that it reduces page utility after accounting for the results that would replace it. The proposed distinction is a risk requirement on the eligibility decision. We did not demonstrate that a separate implementation is necessary or superior to an integrated one. The MSLR cost of 0.10 is a sensitivity parameter in units of normalized relevance gain, representing a stylized cost of filling a slot. It is not a measured business cost or a penalty for false suppression. MSLR and the online simulation provide full-information evidence; Criteo and Open Bandit do not test finite-slot ranking.

Again, thanks a lot for reviewers' effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.