acceptodds
Under review as a conference paper at ICLR 2027

The Information Cost of Exposure-Weighted Risk Certification

Abstract

A risk certificate supports deployment only if it can approve a safe policy with the available data. This requirement is difficult to assess when inputs generate different numbers of predictions or amounts of financial exposure. We study the sample cost of certifying the ratio of expected erroneous exposure to expected exposure. For fixed mean exposure and a specified safe alternative, we reduce the finite-sample minimax problem over bounded observation laws to an exactly solvable three-point experiment. The resulting sample requirement has order , where is normalized mean exposure, is risk, is the risk cap, and bounds false certification, with and target power fixed. This benchmark distinguishes necessary data from conservative test design and loose sufficient bounds. In one zero-risk setting, the optimum requires 27 observations, an implementable sampling test requires 29, and its sufficient bound gives 360. We also connect sampling likelihood ratios to direct product betting and identify the same information scale in worst-case log growth. Controlled experiments and multilabel prediction tasks show when direct tests use population structure that sampling discards, and when sampling remains more effective. Together, the analysis and experiments explain how a reduction can preserve worst-case power while leaving room for improvement on particular populations.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
distribution-free risk certification, sample complexity, selective risk

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 2 reviews, above 9% of the others.
Reviewer_vx9aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the information cost of certifying an exposure-weighted risk. The central risk is the ratio between expected erroneous exposure and expected exposure, which differs from averaging per-example error rates. The paper asks how many fresh observations are needed to certify a policy with high probability when the policy has a specified safe risk, exposure mean, and target risk cap.

The authors reduce the bounded exposure-risk problem to a three-point experiment and derive an exact finite-sample minimax certification benchmark. They compare the optimal budget with an implementable exposure-sampling test and with a sufficient bound. The paper further relates exposure sampling to direct product betting, analyzes worst-case information growth, and studies policy selection on controlled populations, multilabel datasets, and a lending task.

Critical strengths
  • The paper provides an exact and computable worst-case benchmark for the finite-sample cost of exposure-weighted risk certification, rather than relying only on conservative sufficient bounds.
  • The combination of controlled populations, direct-test comparisons, multilabel policy selection, and lending analysis illustrates when exposure sampling is effective and when retaining population structure can improve certification power.
Critical weaknesses
  1. The scope of the minimax benchmark is narrower than the practical certification problem. The exact optimum assumes a fixed policy, known mean exposure, and a specified safe alternative risk. Practical certification often involves unknown exposure means, multiple candidate policies, candidate ordering, and joint constraints. The paper distinguishes some of these settings, but the relationship between the oracle benchmark and the final deployable procedures could be made more explicit.

  2. The theoretical positioning relative to classical testing and sequential risk-control methods could be clearer. The paper connects conformal risk control, LTT, bounded-mean testing, e-values, product betting, and mean-preserving randomization, but it does not fully clarify how its exact minimax result differs from existing minimax testing or sample-complexity results. The distinction between fixed-budget certification and sequential or anytime-valid certification also deserves a more systematic discussion.

  3. The empirical validation has limited breadth and external validity relative to the scope of the claims. The controlled simulations are informative and carefully designed, but the real-data experiments rely mainly on two multilabel datasets and one retrospective lending dataset. Candidate policies are constructed from a relatively restricted model and threshold/gating family, with limited evidence across alternative model classes, exposure-cap specifications, policy families, external datasets, and distribution shifts. The calendar-separated lending results show empirical risk exceedances, but the paper does not systematically study how certification should be adapted under such population changes.

  4. The analysis of practical design choices and failure modes remains incomplete. The paper compares several tests and policy-selection procedures, but the effects of the minimum-count guard, coefficient grids, candidate-family size and order, exposure-cap misspecification, and empty certified sets are not fully isolated. The practical gap between the oracle optimum, the implementable test, and the selected policy under unknown parameters could also be analyzed more extensively.

Questions for authors

Please see Weaknesses.

Reviewer_4qPUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper studiess the sample complexity to certify that a policy’s exposure-weighted risk falls below a threshold α\alpha, subject to strict false-positive (δ\delta) and false-negative (β\beta) error bounds. The authors reduce this certification problem in a minimax sense to a three-outcome statistical experiment, identifying the optimal finite-sample test when the mean exposure μ\mu and true risk rr are known. Through this reduction, they establish an exact sample-complexity bound of Θα,β(log⁡(1/δ)/[μ kl(r,α)])\Theta_{\alpha,\beta}\bigl(\log(1/\delta)/[\mu\,\text{kl}(r,\alpha)]\bigr). The authors accompany their theoretical results with exposure-sampling test that achieves this optimal rate without requiring prior knowledge of μ\mu or rr. Finally, the authors connect these findings to direct betting tests, demonstrating both theoretically and empirically that while no method can uniformly surpass the minimax baseline, retaining raw exposure data can improve sample efficiency when certifying specific subpopulations.

Critical strengths
  1. The authors show a reduction to a three-outcome experiment that identifies the optimal worst-case certification power for any given sample size, establishing a much stronger benchmark than relying on standard asymptotic rates.

  2. The upper and lower bounds clearly delineate how certification difficulty scales with the mean exposure, the margin from the risk threshold, and the acceptable false-certification probability.

  3. Through theoretical and empirical analysis, the authors successfully distinguish fundamental statistical barriers from conservatism in test design, explaining exactly when direct tests can exploit data that exposure sampling discards.

Critical weaknesses
  1. The optimal test relies on knowing the mean exposure and a specified safe alternative risk, meaning its exact finite-sample optimality under unknown parameters remains unresolved even though practical tests match the overall sample-complexity order.

  2. The empirical evaluation is limited by its reliance on two relatively small datasets with repeated data reuse.

  3. The presentation is hard to follow and the main text needed rewriting.

Questions for authors
  1. How informative is the oracle benchmark when mean exposure is unknown?

  2. Can the advantage of direct testing be characterized in finite samples? The likelihood-ratio identity and worst-case log-growth result do not establish a power ordering between direct betting and exposure sampling. Can you provide explicit conditions under which one method requires fewer observations, and clarify whether the second-moment sufficient bound captures the actual improvement?

Comments on the paper

1 comment
Area_Chair_rp6NOpenReview ↗

Review #3

Paper Summary: At a high level, the paper focuses on certification cost for expected erroneous exposure vs expected exposure. The goal is to draw the connection between risk control for selective prediction and statistical tests of bounded observations. This is a mathematical paper on information cost. They contribute a computable benchmark for certification cost. And they offer a comparison between sampling and direct evidence. Sections 3 and 4 contain the bulk of the contribution. Their analysis seems to offer a method for interpreting risk certification.

Strengths

  • I like how narrow the focus is as well as how clear the contribution is. They are careful not to overclaim by my read. And quite nicely situated in the literature though I would have expected a much longer reference list, I have seen these shorter lists for other theory papers.
  • The theory flows nicely. the minimax result, for instance, in Theorem 2 is nice.
  • The work is motivated by a real problem on validation of certificates. Focusing on whether a safe policy can pass with the data given is useful for the research community. This is clear from the introduction and the overall motivation for exposure-weighted risk analysis

Weaknesses

  • Its possible that the paper fits more in a venue like AI Stats and therefore may be challenging for ICLR reviewers to engage with. Not a knock on the submission but worth noting that I am keeping this in mind.
  • while the novelty is clear and carefully stated, the contribution is targeted such that it could be seen as somewhat small
  • I think it is okay for a mainly mathematical paper, just noting that the experiments could be strengthened.

Questions (see weaknesses)

Rating: Weak accept

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.