1. The stated objective is not evaluated, and the experiments as designed cannot show the claimed benefit
(a) The stated objective is not evaluated: false-suppression risk is never reported
Section 3.3 defines the false-suppression (FS) risk RFS(π)=P{h(z)=0, Δ(z)≥0} and the harmful-impression (HI) risk RHI(π)=P{h(z)=1, Δ(z)<0}, and states the objective as high utility while controlling the FS risk. The experiments instead report "violation" Prn(Γ>Uα) (Table 1 caption; Appendix C.1), and only for conformal gates. This column equals ≈α by construction (it is the empirical coverage of the conformal bound) and therefore carries no information about the gate's performance: it is 9.8% both for CoCo-1, which suppresses nothing, and for CoCo-M, which suppresses 39.4% (Table 7). The quantity the formulation asks for, Prn(Uα<0, Γ≥0) (or the same event with Δ in place of Γ), does not appear in any table.
This omission cannot be attributed to identifiability. On MSLR the target is full-information net utility, so Γ=Δ is known for every test document. In the online simulation the latent Δ is known, and the paper already computes Prn(Δ>Uα) (Table 2) while noting in Appendix C.7 that {Uα<0,Δ≥0}⊆{Δ>Uα}. In both cases the FS rate can be computed from quantities the paper has already computed. The version with Γ, Prn(Uα<0, Γ≥0), is computable wherever violation is, including Criteo and Open Bandit.
The same applies to the point gate. Section 6.1 states that "Point's false-suppression risk was not recorded, precluding a matched-risk comparison." This is precisely the comparison the paper's motivation calls for. From the paper's own numbers one cannot rule out that Point already satisfies the risk budget. For any gate, as fractions of all test documents,
FS−HI=suppression rate−fraction of negative candidates,
which for Point at cost 0.10 is 89.7%−83.9%=5.8% (Tables 7 and 5; this assumes the suppression rates in Table 7 are over all test documents, which the paper does not state). If Point's HI rate is small — plausible for a regression fit to a target that is 83.9% negative — then Point's FS rate is within α=0.10, in which case CoCo-M is a gate that satisfies the same risk constraint but has lower utility (0.2848 vs 0.3104, Table 1). The paper should report the FS and HI rates for all gates including Ranker and Point, and compare utility at matched FS.
(b) The motivation is not encoded in the evaluation metric, so the experiments cannot show the claimed benefit
Section 1 motivates conservative suppression by the irreversibility of suppression: a suppressed candidate cannot be recovered by the downstream ranker, whereas an eligible candidate is still subject to ranking. This is an argument that false suppression is more costly than harmful impression. The utility V(π) of Section 3.3, however, counts both error types in the same units. Under a symmetric utility, and up to interactions between candidates, the gate that maximizes expected utility is 1{Δ(z)≥0}; with an estimate Δ^ that is unbiased for Δ, this is the point gate, and a one-sided margin −qα can improve utility only insofar as it corrects bias in Δ^. This is consistent with Table 8 (CoCo-M utility rises monotonically toward Point as α grows) and Table 11 (CoCo-M exceeds Point only at α=0.40, where the margin is essentially zero and the remaining effect is per-group bias correction). As designed, the experiments therefore cannot demonstrate the benefit the motivation claims, and the observed ordering Point > CoCo in utility is the expected outcome rather than a surprising negative result. Either an asymmetric cost for false suppression should enter the objective, or the comparison should be made at matched FS (see (a) above).
(c) Missing obvious baselines
- A tuned-threshold point gate 1{Δ^(z)≥τ} with τ chosen on the calibration split to match a target FS (or suppression rate). CoCo-1 has exactly this form with τ=qα; without this baseline the contribution of conformal threshold selection cannot be separated from the effect of thresholding at a nonzero value.
- On MSLR, the oracle gate 1{Γ≥0} with the same ranker. It is computable in a full-information benchmark and gives the utility upper bound of gating; without it the reader cannot tell what fraction of the achievable gain CoCo-M's +0.0295 represents.
2. Unspecified design choices and an overstated contribution statement
(a) CoCo-M's group definition is unspecified, and CoCo-1 is degenerate
- Section 4.1 lists possible group rules for CoCo-M ("query type, user segment, item type, or a score bin fixed before calibration"), but the group G(z) actually used is stated only for the online simulation ("the three context groups", Appendix C.7). For MSLR and Criteo it is not specified anywhere. The main MSLR result (CoCo-1: 0.0% suppression, CoCo-M: 39.4%, Table 7) therefore depends on an undocumented design choice. If G is a bin of Δ^, group-conditional validity has a quite different meaning from the interpretation in terms of query types, user segments, or item types suggested in Section 4.1. The authors should state what G is and confirm that it was fixed before seeing the calibration data.
- CoCo-1 at α∈{0.05,0.10} suppresses 0.0% of candidates (Tables 7 and 8) and is therefore numerically identical to the ungated ranker in Table 1. The claim that "CoCo-M improves on global CoCo-1 at the conservative budgets" (Figure 1 caption) thus reduces to the statement that suppressing some candidates is better than suppressing none, and does not support groupwise over global calibration. The jump of CoCo-1 from 0.0% to 64.7% suppression between α=0.10 and 0.20 (Table 8) suggests that the global quantile falls at a discontinuity induced by the discrete net-utility target; this should be discussed.
(b) The contribution statement overstates what is new relative to prior CoCo work
The method name, the three variants CoCo-1/M/F with the same global/group/local structure, and the idea of conformal confidence bounds on uplift predictions for content suppression appear in a published SIGIR 2026 short paper ("CoCo: Conformal Confidence Suppression to Optimize Search Results," DOI 10.1145/3805712.3809847), which is not cited. The second contribution, "we introduce CoCo-1, CoCo-M, and CoCo-F, three conformal gates" (Section 1), is therefore not accurate as stated. The actual difference with respect to that work is real but narrower: an explicit calibration target Γ (and the acknowledgement that coverage of Γ is not a guarantee for Δ), groupwise quantiles for CoCo-M (the earlier CoCo-M uses a pooled quantile of group-normalized residuals), a fitted scale model instead of an ensemble for CoCo-F, the DR option, the separation of gate from display, the proofs, and evaluation on public benchmarks. The paper should cite the earlier work and state this difference explicitly; my assessment of novelty above is based on it.
Response to all Reviewers
We thank the reviewers for their time and detailed feedback. The reviews identified gaps in how we evaluated the stated objective and described the calibration protocol. We decided to withdraw the submission to address these issues. Before withdrawing, we would like to respond to the main questions.
False suppression, harmful eligibility, and the Point comparison (to Reviewer A6XX). We agree that bound exceedance is not the false-suppression rate. The relevant events are suppressing a candidate with a nonnegative target and keeping a candidate eligible with a negative target. The latter is an eligibility error; it need not result in display. On MSLR and in the online simulation, the benchmark target or latent utility is available, so the omission was an evaluation gap. For example, the DR Point gate’s 89.7% suppression rate and the 83.9% negative-target share on MSLR imply that false suppression exceeds harmful eligibility by about 5.8 percentage points. These aggregates do not determine whether Point meets the 10% budget. We therefore cannot answer from the submitted results whether CoCo achieves higher utility at matched false-suppression risk. We agree that the revision needs direct error rates for every gate, a threshold chosen on held-out data for the same risk budget, and an MSLR oracle comparison. CoCo-1 is itself a global threshold rule. At the two smaller MSLR budgets it suppresses no candidates, so the comparison to it does not by itself establish the benefit of groupwise calibration.
On Criteo, the archived DR results give false-suppression and harmful-eligibility rates of 1.93% and 74.15% for Point, and 1.11% and 78.77% for CoCo-F at α=0.10, measured against the DR pseudo-outcome. These are target-level diagnostics. Randomized treatment helps identify conditional mean effects, but the sign of a noisy pseudo-outcome does not identify the sign of an individual or conditional causal effect.
Groups and implementation (to Reviewers A6XX, GWZT). In the archived MSLR runner, CoCo-M bins predicted uplift using up to five quantile groups. In Criteo, it bins predicted control conversion risk, also using up to five groups. In both runs, the cut points were estimated from predictions on the calibration split and then applied to the test split. They were not fixed on a separate split before calibration. The MSLR runner also fits the CoCo-F scale using calibration residuals. We should have documented these choices and distinguished those empirical runs from the split-calibration conditions used in the theory. For deployment, our starting choice would be CoCo-1 for a pooled risk target, CoCo-M when meaningful groups can be specified in advance and have enough calibration data, and CoCo-F when residual scale varies with context and can be fit on separate data. We have not established that one variant is uniformly preferable.
Presentation, rankers, and theory (to Reviewer GWZT). In Section 4.1, βh denotes the fitted parameters of the outcome model for arm H=h. We should define it at first use and explain the sequence from fitting uplift, to constructing the calibration target, to choosing the eligibility rule. NDCG@10 measures discounted relevance in the displayed list relative to the ideal top ten from all candidates for that query. It excludes display cost and is a separate ranking-quality metric. The false-suppression theorem does not imply an NDCG guarantee. Appendix C.5, Table 7, tests relevance-trained and net-utility-trained LambdaRank models. The qualitative utility ordering is similar for both, but they share an architecture. Proposition 4 is a regret decomposition under an additional value-loss assumption. The current analysis does not establish sublinear online regret or a utility advantage from calibration.
Practical role and evaluation scope (to Reviewers A6XX, w7ht). A ranker with a no-display option can implement the same risk constraint if it has the necessary uncertainty estimates and can enforce the constraint. A separate gate is useful when an existing ranker allocates slots by relevance and a different decision determines whether a module should be eligible based on its incremental effect. For example, an eligibility rule can withhold a shopping module when there is enough evidence that it reduces page utility after accounting for the results that would replace it. The proposed distinction is a risk requirement on the eligibility decision. We did not demonstrate that a separate implementation is necessary or superior to an integrated one. The MSLR cost of 0.10 is a sensitivity parameter in units of normalized relevance gain, representing a stylized cost of filling a slot. It is not a measured business cost or a penalty for false suppression. MSLR and the online simulation provide full-information evidence; Criteo and Open Bandit do not test finite-slot ranking.
Again, thanks a lot for reviewers' effort!