acceptodds
Under review as a conference paper at ICLR 2027

The Information Cost of Asymmetric Risk Control

Abstract

When prediction errors have unequal consequences, overall accuracy may conceal failures that warrant separate risk limits. We ask how the granularity of these limits changes the information a representation must retain. We study finite-message representations fixed before numerical risk budgets are specified, with frontier error defined as the largest increase in optimal prediction error across all budgets. For a regular class of classification problems with a common default that avoids severe errors and one competing action, we prove that grouping the same severe error events into groups yields optimal -message frontier error . With the source and error events held fixed, finer control thus changes the rate at which additional messages recover decision performance, even when each grouping receives its own optimal representation. Separate limits make distinctions between error types relevant to action; we identify cases where proportional risks or a default action that is optimal at every input eliminate the added cost. Preserving the frontier further bounds the loss in optimal objective value when risk preferences are specified after encoding. Synthetic experiments illustrate these effects, while experiments on frozen, calibrated WiCE verifier outputs show a corresponding cost of separate overclaim control over the tested penalty family.

Primary area:
learning theory
Keywords:
asymmetric risk control, evidence reasoning, evidence verification, representation complexity, constrained classification, quantization

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_dSUzOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

-the authors analyze how the grouping of asymmetric error constraints changes the amount of information a compressed representation must retain. -for a structured class of classification problems, it proves that the optimal frontier error for an M-message representation scales as Theta(M^(-2/m)) when severe errors are grouped into m risk groups. -paper also connects frontier preservation to downstream choice of risk preferences and provides synthetic experiments plus experiments on frozen calibrated WiCE verifier outputs.

Critical strengths

-technically substantive paper with a clear conceptual contribution. The main theorem is crisp and the setup is well motivated -experimental section is appropriate for the paper’s goals. The synthetic studies are well aligned with the theory, and the WiCE experiment is not a full real-world benchmark but does provide a meaningful bridge

Critical weaknesses

The theory is derived in a fairly specialized setting with a common default action and one competing risk-incurring action. The paper is honest about this, but the practical scope of the result is narrower than the framing may initially suggest. Also, the empirical section is mainly illustrative rather than validating a broad applied claim. That is acceptable for a theory paper, but readers looking for evidence in learned end-to-end systems may find the practical takeaways somewhat indirect.

Questions for authors

How much of the rate phenomenon do you expect to survive once there are several non-default risk-incurring actions with interacting decision boundaries?

In the WiCE experiment, how sensitive are the qualitative conclusions to the specific verifier and calibration recipe used to produce the frozen posteriors?

1 comment

Thank you for your careful review and helpful suggestions. We respectfully clarify the scope of our results and summarize the additional evidence below.

W1/Q1: What survives with several risk-incurring actions and interacting decision boundaries?

Response: For a fixed encoder, finer control still cannot reduce the worst optimal penalized-risk loss from compression. Merged control ties penalty weights within groups, whereas finer control enlarges the penalty family. This ordering holds with multiple competing actions.

The sharp rate additionally depends on the decision geometry. Theorem 4.2 obtains 2/m2/m through the common gain Δ\Delta and the mm-dimensional state Bw/ΔBw/\Delta. Beyond this structure, we expect rate separation when refinement exposes additional independent distinctions that affect optimal decisions. With multiple actions, some boundaries may be hidden by a better competing action, so identifying the jointly active distinctions is essential. This gives a concrete basis for expecting the mechanism to persist, while the general sharp exponent is not established by our current analysis.

W2: How do the results connect to practical risk control?

Response: In an evidence-verification service, an operator may tighten the allowed rates of labeling unsupported or partially supported claims as “supported.” For a chosen grouping, the verifier and codebook stay fixed; a downstream policy maps the messages to decisions under the new budgets. The practical objective is to preserve these later choices when designing the representation.

We tested this workflow using policies selected on development labels, with budget margins chosen by source-group cross-validation. On the official WiCE test set, all 12 configurations—full or M=32M=32 outputs, two groupings and three budgets—met their empirical caps while retaining nonzero supported predictions. This provides label-based evidence that budget adaptation can work with unchanged codebooks, rather than requiring a new representation for every operating point.

Q2: How sensitive are the conclusions to the verifier and calibration?

Response: The qualitative grouping effect was consistent across the tested settings. We compared uncalibrated probabilities with temperatures T^\hat T, 0.8T^0.8\hat T and 1.2T^1.2\hat T, where T^\hat T is the fitted temperature. Codebooks were refitted for each setting, and fixed messages were decoded using their development-set mean posteriors. Split control had higher worst posterior-based regret in all 48 comparisons across the two original verifiers, four temperature settings and M∈{4,8,16,32,64,128}M\in\lbrace 4,8,16,32,64,128\rbrace.

Two additional, separately calibrated verifier pipelines gave these split-to-merged regret ratios:

Verifier pipelineM=8M=8M=32M=32M=128M=128
DeBERTa-large2.664.807.56
ModernBERT2.2010.0714.61

The consistent ordering supports a grouping effect beyond the original verifier and calibration choice. The varying ratios show that its magnitude remains dependent on the verifier pipeline.

Reviewer_PeGuOpenReview ↗
Rating
4(+1)
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work studies how the grouping of severe errors changes the information needed to preserve decision performance. They show that preversing separate risk budgets require finer representation, hence increased information cost, to compensate degradation in decision accuracy. They provide necessary theoretical framework as well as synthetic and real world experiments.

Critical strengths
  1. The paper first provides theoretical framework, then show their results both on synthetic experiments and more realistic setups with a frozen evidence verifier. By doing so, they provide a complete picture.
Critical weaknesses
  1. The paper is hard to follow for someone unfamiliar with statistical decision theory. While authors put some effort with nicely chosen examples, I believe the language throughout the paper can still be simplified. At least, explaining WiCE experiment results and its relationship to theoretical framework can be grounded into terminology of that application area for clearer narrative.

  2. One issue regarding WiCE setup, it is a simple task where even finer representations (M=128) are not a burden for deployment with current hardware. The granularity of representation is decided during development/training stages and the paper does not show that dynamic. Maybe working quantized models in a different setting might constitute a more realistic example while still satisfying the authors' assumptions.

Questions for authors

No additional question.

1 comment

Thank you for your careful review and helpful suggestions on making the practical connection clearer. We respectfully clarify the interpretation and use of our results below.

W1: Can the theory and its practical use be explained more clearly?

Response: Consider a service that checks whether evidence supports a claim. It may limit all incorrect “supported” judgments together, or set separate limits for claims with no support and claims with only partial support. With separate limits, knowing only the total chance of overclaiming may not be enough: the decision also depends on which mistake is more likely. Our theory measures how many distinct messages a compressed representation needs to preserve decision quality across different budget choices. It shows that separate limits can require more messages, even when the data and error types stay the same.

The practical lesson is to decide which errors need separate protection before building the codebook, and check the compressed outputs across the budgets users may later choose. WiCE illustrates why this matters: with the same message limit, separate control loses more decision quality through compression. This connects the theory to a concrete design choice—what information to keep for later risk decisions.

W2: If M=128M=128 is inexpensive and fixed during development, what is the practical benefit?

Response: We agree that 128 messages do not impose a substantial hardware burden and that the codebook is chosen during development. Here, MM counts the possible messages passed to the decision rule; 128 messages can be identified by a seven-bit index. Our question is whether those messages retain enough information for later risk choices. Model-weight quantization addresses the complementary cost of storing and running the verifier.

For a fixed grouping, users can later tighten or relax the allowed error rates. The system responds by changing how messages are mapped to decisions, while keeping the verifier and codebook fixed.

We tested this with one fixed M=32M=32 merged-control codebook. Decision rules were selected on development data, with budget margins chosen by source-group cross-validation. The official test results were:

Total overclaim capObserved overclaim ratePredicted “supported”
4.725%3.18%46.73%
7.088%6.37%54.27%

Rates are percentages of all test examples, averaged over policy randomization. Under the stricter cap, the same messages produce fewer overclaims and fewer “supported” outputs. Both policies meet their empirical caps. This demonstrates different risk–output tradeoffs using an unchanged verifier and codebook.

Reviewer_FrkQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how grouping severe prediction errors affects the size of a representation that must be fixed before risk budgets are chosen. Under assumptions that include known true posteriors and two competing actions, the authors prove that the optimal frontier error with mm risk groups scales as Θ(M−2/m)\Theta(M^{-2/m}), where MM is the number of messages. They also give cases in which finer grouping adds no cost. The authors evaluate these results using synthetic data and calibrated outputs from a WiCE evidence verifier.

Critical strengths
  1. The main theoretical comparison is clear. The authors keep the data distribution and error events fixed while changing how the errors are grouped. They prove different error rates even when each grouping has its own optimized encoder. The lower bound also covers randomized encoders.

  2. The examples help define the result’s scope. The authors show that finer grouping need not increase the number of messages when risks are proportional or the default action is always optimal.

Critical weaknesses
  1. The WiCE experiment provides limited evidence about actual risk control. The theory assumes known true posteriors and a specific two-action structure, but the paper does not establish that these conditions hold for WiCE. Figure 4 measures loss using model probabilities rather than development-set labels, so it does not show the observed overclaim rates or whether risk budgets are met.

  2. The experiments leave some uncertainty about the source of the reported gains. In Figure 2, the k-means methods differ in both the variables they cluster and the weights they use. In Figure 4, decisions use fitted codewords, whereas the theoretical measure allows optimal decisions given each message. Additional comparisons would help separate the effect of risk grouping from fitting and decoding effects.

Questions for authors
  1. Could the authors report classification error, the two overclaim rates, and risk-budget feasibility using the WiCE development-set labels? Uncertainty estimates across source groups and calibration results would also help interpret the findings.
  2. How much of the Figure 4 gap remains if the authors decode each message using its mean posterior on the development set rather than its fitted codeword?
  3. It would be helpful to know whether the Figure 2 advantage remains when posterior k-means also uses Δ\Delta weights and both methods use multiple initializations.
1 comment

Thank you for your careful review and helpful suggestions for separating grouping, fitting and decoding effects. We respectfully clarify the scope and address your three questions and the related weaknesses below.

Q1 (also W1): What do development-set labels show about risk control, uncertainty and calibration?

Response: The label-based evaluation shows that fit-selected policies exceed the dev budgets, including with full outputs. A separate held-out evaluation with validation-selected budget margins achieves empirical feasibility across all 12 tested configurations. The synthetic experiments test the sharp rate under the stated true-posterior and two-action assumptions. WiCE examines the grouping phenomenon with estimated posteriors and three prediction actions.

For the full-evidence verifier, policies selected on the fitting split yielded the following label-based results on 949 development examples. The merged cap is 4.725% in total; split caps are 2.3625% per event. Rates are percentages of all examples, averaged over policy randomization.

ControlRepresentationN→SN\to SP→SP\to SClassification errorExcess (pp)
MergedFull outputs3.79 [2.62, 5.09]6.17 [4.55, 7.98]20.78 [17.98, 23.56]5.24
MergedM=32M=323.83 [2.65, 5.13]6.05 [4.39, 7.86]21.07 [18.25, 23.97]5.15
SplitFull outputs3.60 [2.46, 4.87]5.77 [4.23, 7.48]20.78 [18.00, 23.54]3.41
SplitM=32M=323.27 [2.17, 4.54]5.37 [3.82, 7.04]20.65 [17.82, 23.32]3.01

Brackets give pointwise 95% source-group bootstrap intervals for frozen policies; excess is the largest applicable budget violation. Both full and compressed outputs exceed the dev caps, so these violations are not solely due to compression.

We also evaluated a practical selection procedure. Keeping the verifier and codebooks fixed, we selected policies on dev labels with budget margins chosen by source-group cross-validation, then froze them before evaluation on the official 958-example test set.

All 12 configurations—full and compressed outputs, two groupings and three budgets—met their empirical caps with nonzero supported predictions. Under the 4.725% merged cap, for example, M=32M=32 achieved 3.18% total overclaim, 22.73% classification error and 46.73% supported outputs. These results show that fixed messages can support subsequent budget choices while retaining useful supported predictions.

Calibration results on dev are below; temperatures were fitted on a separate 171-example calibration split.

VerifierNLL: raw → calibratedBrier: raw → calibratedTop-label ECE: raw → calibrated
Full-evidence0.552 → 0.5400.308 → 0.3030.054 → 0.022
Neighbor-masked0.612 → 0.5970.349 → 0.3410.066 → 0.022

All three metrics improve for both verifiers.

Q2 (also W2, Figure 4): How much of the gap remains with development-set mean-posterior decoding?

Response: The split-control loss remains larger. For each grouping, we kept its message assignments fixed and replaced each fitted codeword with the mean model posterior among development examples assigned to that message. At M=128M=128, the worst posterior-based regrets change as follows, in units of 10−410^{-4}:

VerifierMerged: fitted → meanSplit: fitted → mean
Full-evidence1.236 → 0.7666.732 → 4.926
Neighbor-masked2.063 → 0.96811.753 → 11.216

After re-decoding, split losses remain 6.43 and 11.59 times the merged losses. Across both verifiers and six codebook fits, split loss is larger in all 72 comparisons with M≥4M\ge4; the twelve comparisons at M=2M=2 tie. Thus, fitted codewords contribute to the original loss, but the grouping difference persists under optimal decoding for the empirical posterior distribution within each message.

Q3 (also W2, Figure 2): Does the advantage remain with matched Δ\Delta weights and multiple initializations?

Response: Yes. We used Δ\Delta weights in both clustering spaces, ten initializations per fit, and selected each codebook by its lowest fitting objective. Both methods used message-conditional decoding and three independent Sobol evaluation sequences.

At M=4096M=4096, the ratios of mean posterior-k-means loss to mean decision-state-k-means loss were 6,247, 16.85 and 1.05 for one, two and three risk groups, respectively. The advantage for one and two groups therefore persists after matching weights and initialization budgets. With three separate groups, the losses are nearly equal, consistent with the absence of dimensional reduction in that case.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.