acceptodds
Under review as a conference paper at ICLR 2027

When More Candidates Hurt: A Proposal–Evidence Reliability Frontier for Causal Selection

Abstract

Candidate generation can scale faster than causal evaluation. Under a fixed evidence pool, enlarging a candidate prefix changes both the best attainable action and the statistical burden of comparing candidates with uneven causal support. We characterize this joint movement through a *proposal–evidence reliability frontier*. Proposal-Certified Selection (PCS) uses proposal-side predictions to order candidates and lower endpoints of simultaneous causal intervals to select among them. Its all-prefix certificate balances comparator miss against certification width across all covered prefixes. In a Gaussian benchmark, a lower bound yields a comparison requirement, showing that logarithmic multiplicity persists beyond the plug-in selector. A proposal/support-aware allocation uniquely minimizes the stated expected certificate-width objective; realized decision performance is evaluated separately. Controlled diagnostics reproduce the predicted selection-error and multiplicity patterns. In Upworthy replays, the paired PCS gain over tuned fixed depth rises from 0.042 to 0.949 basis points as evidence increases from 0.5% to 5%. In a 26.9M-impression recommendation-log study, the randomized-audit click-through-rate (CTR) gain over inverse-propensity-score (IPS) plug-in is 26.70 basis points (95% cluster interval ). The ten-issue persuasion study remains unresolved, and uniform allocation nearly matches adaptive allocation in recommendation. Together, these results separate proposal coverage from statistically usable decision improvement.

Primary area:
learning theory
Keywords:
language-action sets, causal selection, policy selection, logged feedback, off-policy evaluation, statistical decision theory, uncertainty quantification, multiplicity, Best-of-N

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_q6swOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Proposal-Certified Selection, a technique for selecting among multiple proposals in causal selection. The paper first shows that under noisy estimates of causal value, having more proposed candidates can hurt the expected regret under a simple plug-in estimator. Then, they propose a framework called PCS for selecting a candidate and validate it theoretically and empirically.

Critical strengths
  • Interesting and important problem setting

I think the problem setting is interesting, and there could be value to the framework you are proposing. I encourage you to keep working on this topic and improve the paper and results.

Critical weaknesses
  • Paper is very poorly-written
  • Main theoretical results are very modest, some following trivially from assumptions or common knowledge

The paper in its current form is far from the bar for publication, in my view. This paper was very difficult to read and appears to be substantially written by AI. The abstract, introduction, and related work seemed completely AI-written with confusing logical structure, undefined terms, and convoluted sentence structure. The confusing writing appeared throughout the rest of the paper, to varying degrees. For instance, in 3.2, what does “The order may use generation randomness, retrieval, pretrained scores, or disjoint historical outcomes” mean? The order of what? The concept of G\mathcal G is not precisely defined anywhere, and it’s not clear what you mean by “Let G\mathcal G contain proposal randomness, fitted objects, the realized sequence, and pre-outcome design metadata.” The appendices largely read as if they were AI generated (even if not, they were very convoluted and difficult to read).

In Section 3, the “Notation” paragraph is duplicating most of the information in 3.1, except Section 3.1 more clearly explains what your notation means. I suggest removing this redundancy. (With the exception that 3.1 does not define the value estimates ν^\hat \nu).

Several of the main theorem statements are obvious or tautological. Thm. 1 is a very basic observation and not at the level of a theorem, in my view (it could also be significantly generalized beyond Gaussians, but I still think the key observation would not be interesting). The Theorem also does not specify what the “plug-in selector” is. Later theorems (e.g., Thm. 2) are tautological based on the definition of the selector, or follow directly from restrictive assumptions on the underlying random process (Thm 4). Overall, I did not find the theoretical results to be sufficiently novel or technically interesting to meet the bar for publication at this venue.

The figures are almost impossible to read. There were several parts of the figures that I could not read at all.

Questions for authors

Truthfully, I don't believe my main concerns can be resolved through Q&A or running more experiments. However, if you had to pick 1-2 theoretical results that are the "main result", which ones would they be, and what is their principal novelty?

Reviewer_V4zzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors analyze a procedure for policy learning, where a growing set of options is considered and the goal is to characterize the trade-off between "more options" being able to achieve higher population level value, while at the same time "more options" leading to higher sampling error. This creates two opposing tensions and a frontier.

Critical strengths

The question of how you balance "more options" so as to have more opportunities of success, vs sampling error, is a very interesting trade-off and worth studying.

Critical weaknesses

Unfortunately, I found the authors' results very incremental and weak in addressing the posed question. The authors are essentially in the end performing "structural risk minimization" over the space of nested function spaces, defined by prefixes of a fixed order of the options. They use a lower confidence bound strategy, where they allocate the "lower confidence" probabilistic budget across the arms. None of these steps are conceptually novel and combine known tools.

Questions for authors
  1. How do you decide the order of the options?
  2. Why do you use essentially a "union bound" strategy for the lower confidence part. There are better methods for constructing simultaneous confidence intervals that are more adaptive to the covariance of the error of the various options.
  3. Why not simply perform structural risk minimization? What is the big difference between what you propose and SRM?
  4. Is your procedure in any way uncovering an "optimal accuracy-variance" frontier?
Reviewer_m3oVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper considers a setting in which a cheap proposer produces an ordered list of candidate actions, and a fixed pool of randomized or logged data is then used to estimate each candidate’s causal effect. Considering more candidates makes a better action available, but it also increases the chance of selecting a candidate whose effect was overestimated by noise. The paper proposes Proposal-Certified Selection (PCS), a confidence-interval-based selection method that selects the candidate with the highest lower confidence bound (LCB). Specifically, PCS builds simultaneous confidence intervals using a weighted Bonferroni split that allocates more of the error budget to higher-ranked candidates, and then selects the candidate with the highest LCB. On the theory side, the paper shows that selecting the highest point estimate can get worse with every added candidate. It also gives an all-prefix bound showing that PCS regret is at most the best combination, over candidates, of a candidate’s distance from the optimum plus its interval width. Experiments use Gaussian simulations, Upworthy headline A/B tests, an LLM persuasion study, and the Open Bandit Dataset with a randomized audit.

Critical strengths
  • The problem is clearly framed and timely. “Generation scales faster than evaluation” is a real concern for LLM-proposed interventions. Splitting decision error into “the best option was not proposed” and “noise selected the wrong option” is a useful lens.
  • The data are separated carefully. Proposal ordering and tuning are fixed before held-out outcomes are opened, the data are split chronologically, and all methods share the same ordering and the same evidence and holdout partitions.
Critical weaknesses
  • Theoretical novelty is limited. The main bound follows the standard argument for selecting by lower confidence bound, and its validity over all prefixes follows directly from a single simultaneous coverage event. Weighted Bonferroni, the winner’s-curse example, and the Fano-type lower bound are all well-established tools. The paper’s own novelty statement places the contribution in their combination, and it is unclear that this is sufficient.
  • The evidence for the paper’s own component is mixed. The natural baseline is the same rule with an equal error-budget split (uniform simultaneous LCB). PCS improves on uniform LCB in point estimates on Upworthy and the persuasion study, but the margins are small and no paired uncertainty against uniform LCB is reported. In the Open Bandit Dataset, the direction reverses: uniform LCB outperforms default PCS and ties adaptive PCS. In the persuasion study, the confidence intervals for PCS gains over tuned fixed depth all include zero.
  • The guarantee is not instantiated in the largest experiments. The authors acknowledge that the Open Bandit intervals are selection scores rather than valid confidence intervals, and the Upworthy intervals target a different quantity than the held-out outcome. As a result, the theory and the empirical results are only loosely connected.
Questions for authors
  • What are the paired confidence intervals for PCS versus uniform simultaneous LCB in all real-data studies? Without them, the benefit of rank-weighted error spending cannot be assessed.
  • Is there a result in which the growing-prefix structure matters beyond a single coverage event? For example, can the method choose from data how many candidates to consider, or identify when adding candidates stops helping under PCS?
  • In the Open Bandit study, which data were used to compute the variance scale for adaptive allocation? If they came from the same logs used to construct the intervals, the weights depend on the data they are meant to protect, which violates the required separation.
Reviewer_bNSNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Figure 2: The paper studies selection from an ordered, growing set of candidate actions when causal evidence is fixed. Proposal-Certified Selection (PCS) uses proposal information to order candidates and simultaneous lower confidence bounds to select among them. The contributions include an all-prefix regret certificate, a Gaussian comparison lower bound, and an optimal allocation for a specified confidence-width objective, supported by simulations and retrospective studies.

Critical strengths
  • The problem is interesting and well motivated: proposing additional actions can be inexpensive, while obtaining enough causal evidence to distinguish them is costly. The combination of causal inference, simultaneous inference, and policy selection is a particular strength.
  • The theory clearly separates the best action available within a proposal set from the action that can be reliably selected. The all-prefix guarantee and comparison lower bound provide useful perspectives on this distinction.
  • The empirical comparisons include meaningful alternatives, such as uniform simultaneous lower bounds and tuned rank-plus-variance penalties. The support-thinning experiments help isolate the role of uneven evidence.
  • The paper is careful about its claims. It distinguishes confidence-width optimality from better decisions, acknowledges unresolved empirical comparisons, and makes causal identification and interval validity explicit requirements.
Critical weaknesses
  • The opening examples are helpful, but the introduction moves too quickly to “proposal breadth” and a “reliability frontier”, without a more pedagogical/expositional presentation of such concepts. A small numerical example showing how adding a candidate can improve the available options while making the choice less reliable would make the main idea easier to follow.

  • The statistical justification for inference on the conditional value needs clarification. For continuous XX, the value at a fixed xx is generally not pathwise differentiable in a nonparametric model. The AIPW pseudo-outcomes have the desired conditional mean, but they are not influence functions for this pointwise target. The integrated nuisance-rate argument in H.2 therefore does not directly justify inference for the localized estimator in Equation (7). The connection to the conditional bias and concentration assumptions in H.3 should be made explicit.

  • The interval construction also needs a concrete implementation. Appendix H.3 allows localization bias through Bj(x)B_j(x), but leaves the weights and a usable bias bound unspecified. A worked example, including a bandwidth choice if kernel smoothing is used, should explain the bias–variance trade-off and show how to construct intervals satisfying Equation (9).

  • It is still unclear how a practitioner should use the frontier. The paper acknowledges that it depends on unknown value gaps and cannot directly serve as a stopping rule. How, then, does it help answer the motivating question of how many candidates the available evidence can support? An example based on quantities available to the practitioner would be useful.

  • Theorem 7 gives an optimal allocation for the stated width objective, but its usefulness depends on the chosen comparator prior and support scales. The sensitivity analysis shows that a poor prior can lead to worse decisions. More guidance on choosing these inputs and recognizing when adaptive allocation is likely to help would strengthen the result.

  • The results show a clearer benefit from accounting for uncertainty than from the proposed allocation itself. The persuasion comparisons remain unresolved, rank-plus-variance performs better at some Upworthy budgets, and uniform allocation essentially matches adaptive PCS in recommendation. Reporting uncertainty for the differences against the strongest alternatives would make the added benefit easier to assess.

  • The experiments do not fully establish that the intervals meet the coverage requirement used by the theory. In recommendation, the IPS intervals are used as selection scores, and varying their scale does not check coverage. A simulation with estimated nuisance functions and continuous contexts could check whether the intervals cover all candidate values together and whether the regret bound holds across prefixes.

Questions for authors
  • Could you provide one complete continuous-context example for Equation (10), specifying the weights, smoothness assumptions, bandwidth, and construction of Bj(x)B_j(x)? How would nuisance error and localization bias enter the reported intervals?
  • What practical guidance does the frontier provide for choosing proposal breadth when the comparator gaps are unknown? A small worked example would make this contribution easier to assess.
  • How should the comparator prior qjq_j and scales aja_j be obtained in practice? Beyond the existing sensitivity results, can you give conditions under which the width-optimal allocation should improve decisions relative to uniform allocation?
  • Could you add a simulation with estimated nuisances and continuous covariates that checks simultaneous coverage and the all-prefix regret certificate directly? This would help connect the interval assumptions to the proposed estimation procedure.
  • How does the integrated nuisance-error argument in H.2 support inference for the value at a fixed continuous xx, given that this target is generally not pathwise differentiable? What additional local assumptions connect Equation (7) to the bias and concentration bounds in H.3?
  • Could you report paired differences, with uncertainty intervals, between PCS and the strongest alternatives, particularly rank-plus-variance in Upworthy and uniform allocation in recommendation? This would help distinguish the benefit of the proposed allocation from the benefit of accounting for uncertainty more generally.
  • What distinguishes the green triangles from the yellow circles in Figure 2, given that both are labeled as noisy estimates? Could you mark the selected candidate and show how its true value determines the displayed selection gap?
Reviewer_4tTTOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies causal action selection under increasing candidate set sizes and limited evaluation data. It proposes Proposal-Certified Selection, which uses simultaneous confidence intervals for reliable selection, establishes regret bounds, and evaluates the approach through synthetic experiments and three real-data studies.

Critical strengths
  1. The paper addresses an important practical problem: generating more candidates does not necessarily lead to better decisions when causal evidence is limited.
  2. The theoretical analysis provides useful insights into the tradeoff between proposal quality, estimation uncertainty, and candidate set size.
  3. The experiments cover multiple settings and include comparisons with several uncertainty-aware selection baselines.
Critical weaknesses
  1. PCS largely builds on existing lower-confidence-bound selection and weighted simultaneous inference. The main theoretical guarantees also rely heavily on standard concentration and multiple-comparison arguments.
  2. The proposed reliability frontier depends on unknown true causal values and cannot directly determine an optimal candidate depth in practice.
  3. The theoretical guarantees require valid simultaneous confidence intervals, but the OBD experiment uses IPS intervals without establishing the required finite-sample validity.
  4. PCS does not consistently outperform uniform LCB. In OBD, uniform LCB slightly outperforms the default PCS, while adaptive allocation provides almost no additional improvement. The persuasion results are also statistically inconclusive.
Questions for authors
  1. What is the main practical advantage of PCS over standard uniform LCB selection? Besides, can the reliability frontier be used to select candidate depth without knowing the true causal values?
  2. How can the theoretical interval-validity assumptions be verified in real-world applications?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.