acceptodds
Under review as a conference paper at ICLR 2027

Safe Probability Editing Is a Risk Allocation Problem

Abstract

Under distribution shift, adaptation, calibration, and pooling can propose useful probability edits before target labels are available. The deployment question is how a fixed unlabeled cohort should spend a limited risk budget, not only whether every row passes the same pointwise check. Uniform clipping assigns every input the same allowance and therefore ignores large differences in candidate gain per unit risk. We show that an elementary worst-label identity has a consequential operational interpretation: all-outcome regret becomes a conditional-risk currency, turning label-free probability editing into hierarchical cohort allocation. Each row preserves the anchor action and obeys a local catastrophe cap, while a shared ledger distributes fixed worst-case Brier and log-regret budgets across the cohort. Because realized predictive utility is unavailable at edit time, the ledger minimizes intervention from an externally proposed candidate subject to this safety contract. The resulting convex program reduces to two risk prices that decouple the cohort into one-dimensional edits. On 10,000 ImageNetV2 predictions, the same mean budgets and a twofold local cap raise retained update from 19.9% to 30.0% and reduce total squared candidate distance from 176.1 to 123.4, with zero action changes. Across 20 frozen VLCS domain-seed conditions, retained update rises from 14.9% to 22.2% (paired gain 7.30 points; 95% CI 5.94-8.66). Under identical twofold local caps, the ledger achieves the smallest squared candidate-distance objective among five allocation rules. Exact pointwise safe bodies and sparse hulls provide supporting local geometry and scalable extensions. The guarantee is deterministic for the cohort in hand rather than a future-sample population claim. Together, the results show that safety need not mean uniform restraint: risk can be spent where it preserves the most candidate signal, without target labels or action changes.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Safe probability editing, uncertainty quantification, distribution shift, test-time adaptation, risk allocation, calibration, convex optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_upaoOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies label-free probability editing relative to an anchor predictor. It formulates safety through worst-label Brier and log-loss regret, combining per-example caps with shared cohort budgets while preserving the anchor’s decisions. A convex allocator minimizes squared distance to externally proposed candidate predictions and decomposes through two dual risk prices.

ImageNetV2 and VLCS experiments demonstrate increased candidate-update retention, supplemented by predictive-quality evaluations and large-scale numerical safety audits.

Critical strengths
  1. A precise and operational safety formulation.

The method bounds degradation for every possible label while preserving the deployed action. Combining local caps with a cohort budget provides a clear way to permit heterogeneous edits without allowing unrestricted harm on individual examples. This is useful when target labels are unavailable and the proposed updates may be unreliable.

  1. Tractable optimization backed by implementation audits.

The two-price decomposition, analytic ray endpoints, and certified sparse approximations make the formulation computationally accessible. The reported stress tests and explicit treatment of numerical precision and decision ties strengthen confidence that the mathematical constraints can be enforced in practice.

Critical weaknesses
  1. The headline improvement may overstates what the comparisons establish.

The headline improvement uses looser per-example caps, while gains over a strong heuristic under matched constraints are modest. The exact allocator’s added value therefore remains insufficiently demonstrated.

Questions for authors
  1. Which comparison supports the conclusion’s approximately 50% improvement under “matched budgets and caps”? What is the retained-update improvement over the strongest heuristic when both cohort budgets and local caps are identical?

  2. Under those identical constraints, how do the ledger and marginal-efficiency heuristic compare in realized Brier/NLL and computational cost?

1 comment
AuthorsOpenReview ↗

Separating Budget-Matched and Cap-Matched Gains and Quantifying Utility and Cost

Thank you for pointing out the ambiguity in the headline comparison. We agree that the comparator, local caps, cohort budgets, and metric must be explicit. We have revised the Abstract, Section 4.2, and Conclusion. The new Table 3 in Appendix C reports matched-cap results and measured runtimes, with paired predictive-score contrasts reported below the table. References below follow the revised manuscript's numbering.

Q1. What supports the approximately 50% figure? Figure 2a holds the cohort budgets fixed. Pointwise clipping uses onefold row caps and the ledger permits twofold caps. Retained update rises from 19.9% to 30.0% on ImageNetV2 (approximately 50.7% relative) and from 14.9% to 22.2% on VLCS (+7.30 points; 95% CI [5.94,8.66]). This is a budget-matched comparison.

Our additional comparison matches both twofold caps and cohort budgets. Compared with equal-cap-fraction allocation, the ledger raises retained update from 19.962% to 29.955% on ImageNetV2 (+9.993 points, +50.06% relative). The corresponding VLCS comparison is 17.458% to 22.174% (+4.716 points, +27.01% relative). The approximately 50% gain at matched caps refers specifically to the ImageNetV2 comparison with equal-cap-fraction allocation. The revised Abstract reports these cap-matched endpoints; the Conclusion gives the dataset-specific relative gains.

Q2. Improvement over the strongest heuristic. Figure 2b-c and Table 3 compare five rules under identical twofold caps and budgets, all with zero action changes. ImageNetV2 total squared candidate distance is 123.39 for the ledger, 123.91 for marginal efficiency, 123.99 for distance priority, 175.78 for equal-cap-fraction allocation, and 191.32 for confidence priority. The ledger improves on marginal efficiency by 0.52 (0.42%). On VLCS, the paired ledger-minus-marginal difference is -0.346 (95% CI [-0.532,-0.161]) across 20 domain-seed conditions.

Retained Euclidean update is a different metric. Marginal efficiency retains 30.18% versus 29.955% for the ledger on ImageNetV2, about 0.23 points more, and approximately 0.11 points more on VLCS. The global optimality claim applies to the specified squared-distance objective, up to numerical tolerance.

Q3. Predictive utility under matched constraints. Table 3 reports the following measured scores:

Dataset / methodBrierNLL
ImageNetV2 / marginal efficiency.0004105531.342474
ImageNetV2 / ledger.0004100401.340520
VLCS / marginal efficiency.075611.786981
VLCS / ledger.075565.785049

VLCS values are means over 20 conditions. ImageNetV2 ledger-minus-marginal differences are −5.13×10−7-5.13\times10^{-7} in Brier and -.001954 in NLL. VLCS paired differences, computed before averaging, are −4.61×10−5-4.61\times10^{-5} in Brier (95% CI [−6.86,−2.37]×10−5[-6.86,-2.37]\times10^{-5}) and -.001932 in NLL ([-.003198,-.000666]). Both VLCS intervals exclude zero across the 20 paired conditions. Those conditions share four domains, so the intervals describe the observed cohorts. On ImageNetV2, distance priority attains slightly lower Brier and NLL than the ledger. Candidate fidelity and predictive quality therefore require separate evaluation. Appendix C reports the paired contrasts.

Q4. Additional computation. Marginal efficiency uses a priority rule. The ledger optimizes two cohort-level dual prices and solves scalar row minimizers by vectorized bisection at each evaluation. Section 3.2 specifies O(nK)O(nK) preprocessing and O(MnT)O(MnT) allocation for MM dual evaluations and TT bisection steps, followed by numerical feasibility repair.

CohortMarginal efficiencyLedgerLedger / heuristic
ImageNetV20.41 ± 0.03 s1.35 ± 0.08 s3.29×
VLCS, per cohort0.018 ± 0.003 s0.057 ± 0.006 s3.17×

Both methods used the same arrays, caps, budgets, tolerances, and eight CPU threads on an AMD EPYC 7543 (32 physical cores, 64 logical threads) with GPU disabled. The environment was Ubuntu 22.04.5 LTS, Python 3.9.19, NumPy 1.26.4, and SciPy 1.13.1. We used one untimed warmup and ten timed runs per cohort with time.perf_counter. The ± values are sample SDs (ddof=1) across ten ImageNetV2 runs or 20 VLCS cohort means, each averaging ten runs. Timing includes local-cap preprocessing, allocation, bisection where applicable, and feasibility checks and repair. It excludes inference, prediction generation, disk I/O, and loading.

Section 4.2 and Appendix C now document this trade-off. Marginal efficiency may be preferable when latency matters most. The ledger offers the minimum squared distance at additional measured cost. The ratios apply to these implementations and cohorts. We appreciate the reviewer's request to quantify this trade-off.

Reviewer_GvJ2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a hierarchical risk allocation framework for safe probability editing, aiming to preserve candidate predictions while bounding worst-case Brier and log-loss deterioration relative to an anchor model. The formulated convex optimization can be solved using two dual variables, representing the persample and cohort-level risk constraints. Experiments on ImageNetV2 and VLCS show that the proposed method retains more candidate updates than uniform pointwise clipping under the same cohort risk budgets.

Critical strengths
  1. The proposed framework allows flexible risk allocation across samples while maintaining local and cohort-level safety constraints. The resulting convex optimization problem can be efficiently decomposed into independent one-dimensional subproblems.

  2. The experiments cover multiple classification benchmarks under distribution shift. The matched-budget comparisons and numerical stress tests provide evidence for improved candidate retention and satisfaction of the safety constraints.

  3. The paper further characterizes the exact safe probability region and introduces sparse convex-hull approximations, extending safe probability editing beyond a single interpolation direction.

Critical weaknesses
  1. The practical significance of the proposed framework remains unclear. While the method provides theoretical guarantees on worst-case loss deterioration relative to the anchor model, it is unclear how these guarantees benefit real-world applications, particularly in TTA scenarios where the goal is to improve target-domain performance. Moreover, preserving the anchor's predictions by construction prevents the method from retaining any classification accuracy improvements achieved through adaptation.

  2. The experimental evaluation provides insufficient evidence of the practical effectiveness of the proposed method. The main results focus on candidate update retention and distance reduction, while the improvements in actual predictive metrics such as NLL and Brier are limited or statistically inconclusive. More comprehensive comparisons against the unedited candidate predictions and existing adaptation methods are needed to demonstrate whether the proposed risk allocation provides meaningful benefits for target-domain performance.

Questions for authors
  1. In TTA scenarios, the anchor model may perform poorly under distribution shift, and successful adaptation may require changing its predictions. Could the authors clarify the practical significance of enforcing anchor-relative safety and decision preservation, particularly when these constraints prevent improvements in target-domain classification accuracy?

  2. The main experiments demonstrate improved candidate retention, but the gains in actual predictive metrics such as NLL and Brier are limited or statistically inconclusive. Could the authors provide stronger empirical evidence that improved candidate retention translates into meaningful target-domain predictive performance?

  3. Under matched safety constraints, the marginal-efficiency heuristic achieves a squared candidate distance of 123.91, compared to 123.39 for the proposed risk ledger. Given this small difference, could the authors clarify the practical advantages of the proposed optimization framework over simpler allocation strategies?

4 comments
AuthorsOpenReview ↗

Deployment Scope, Realized Predictive Utility, and the Value of Exact Allocation

Thank you for recognizing the tractable allocation framework and for questioning its practical scope. We agree that preserving an anchor decision is a deliberate restriction. The revised Introduction and Discussion clarify this scope, while Section 4.2 and Appendix C report the utility and cost of exact allocation. All table references below use the revised manuscript's numbering.

1. Deployment scope. Our layer operates after a predictor or action policy has been selected. Edited probabilities may be useful for ranking non-top classes and reporting uncertainty while the deployed action remains fixed. The risk guarantee is relative to the anchor. For top-1 classification, holding the action fixed preserves the anchor's accuracy and inherits its errors. When action changes are acceptable and accuracy is the priority, unrestricted test-time adaptation (TTA) or candidate selection may be preferable. For example, on VLCS, temperature scaling plus validation mixing reaches accuracy .7564 versus anchor .7469, but changes 6.61% of decisions (Table 9). We agree that the accuracy gain matters when the deployment permits those decision changes.

2. Value of local safe editing. Table 7 shows a failure case on OfficeHome. Anchor accuracy/NLL is .7662/.9399, versus .7060/1.1964 for the unrestricted candidate, which changes 24.05% of decisions. The Pathwise-Safe Prediction Layer (PSPL) preserves accuracy .7662 while obtaining NLL .9245 and normalized Brier .004964 versus anchor .005014. On VLCS, PSPL preserves accuracy .7469 while NLL improves from .8422 to .7879 and Brier from .07743 to .07603 (Table 9). On ImageNetV2, SafeRay-ViT preserves top-1 .7318 while NLL changes from 1.3620 to 1.3172 and top-5 from .9089 to .9122 (Table 4). These comparisons test the local layer. The separate matched-cap experiment tests the ledger's incremental benefit.

Table 11 gives PSPL-minus-anchor NLL differences of -.01532 (95% CI [-.02142,-.00922]) on OfficeHome and -.05426 ([-.09362,-.01490]) on VLCS. Corresponding Brier differences are -.000050 ([-.000064,-.000035]) and -.001395 ([-.002230,-.000559]). The 20 paired conditions share four domains. Appendix F reports wider intervals using the four domains as the unit. Section 4.3 also reports pointwise NLL gains in 45/45 CIFAR-C corruption–architecture conditions and Brier gains in 44/45. These are empirical results.

3. Incremental ledger benefit. We appreciate the concern that the original approximately 50% figure could obscure the smaller gain over a strong heuristic. That figure compares the ledger with pointwise clipping at equal cohort budgets and different local caps. Under identical twofold caps and budgets, squared candidate distance is 123.39 for ledger versus 123.91 for marginal efficiency on ImageNetV2, a modest 0.52 absolute (0.42%) difference. On VLCS, the paired difference is -0.346 (95% CI [-0.532,-0.161]). Marginal efficiency retains slightly more Euclidean movement on both datasets. The ledger minimizes the specified squared-distance objective under the declared constraints, up to solver tolerance.

The new matched-cap results in Table 3 and Appendix C also show small predictive-score improvements. ImageNetV2 ledger NLL/Brier is 1.340520/.000410040 versus 1.342474/.000410553 for marginal efficiency. On VLCS, paired ledger-minus-heuristic differences are -.001932 in NLL (95% CI [-.003198,-.000666]) and −4.61×10−5-4.61\times10^{-5} in Brier (CI [−6.86,−2.37]×10−5[-6.86,-2.37]\times10^{-5}). The paired intervals describe the observed cohorts and complement the label-free squared-distance objective with a check of realized predictive quality.

4. Computational trade-off. Table 3 reports ledger versus marginal-efficiency runtime of 1.35±.081.35\pm.08 versus .41±.03.41\pm.03 seconds on ImageNetV2, and .057±.006.057\pm.006 versus .018±.003.018\pm.003 seconds per VLCS cohort. The runtime ratios are 3.29× and 3.17×, respectively.

Appendix C specifies the AMD EPYC 7543 (32 physical cores, 64 logical threads) with eight threads and the GPU disabled. Software was Ubuntu 22.04.5 LTS, Python 3.9.19, NumPy 1.26.4, and SciPy 1.13.1. We used one untimed warmup and ten runs per cohort with time.perf_counter. Sample SD (ddof=1) is computed across ten runs for ImageNetV2 and across 20 cohort means for VLCS. Timing includes local-cap preprocessing, allocation, bisection where applicable, and feasibility checks and repair. It excludes inference, prediction generation, loading, and disk I/O. Arrays, caps, budgets, tolerances, and threads are matched.

Section 4.2 now explains why marginal efficiency may be preferable when latency dominates. The ledger is an option when minimizing the declared objective justifies the added cost. Thank you for prompting a clearer separation of safety, fidelity, predictive utility, and computation.

Thank you for the response. My comments follow your numbering.

  1. Deployment scope. I accept that the method is positioned as a post-selection layer rather than as TTA. In that case, its closest relatives are accuracy-preserving calibration methods, which also keep the decision fixed. Please clarify in which applications the additional worst-case guarantee is needed beyond what these methods already provide, since this is the main difference from them.

  2. Predictive value of allocation. The response reports PSPL compared with the anchor, which supports the local layer. My question was about the allocation step. For the ledger compared with uniform clipping, the submission itself states that the Brier/NLL intervals on VLCS "do not resolve a nonzero mean change", and the ImageNetV2 NLL gain is .00251. Table 7 also shows that the full candidate can be worse than the anchor, while a partial move toward it is better, so more retained update does not by itself imply better predictions. The evidence for the main contribution therefore rests on a label-free objective whose link to predictive quality is not established.

3 and 4. Ledger vs marginal efficiency. Thank you for the direct answer. It confirms that under matched caps the ledger gains in the objective are about 3.3x the runtime. The main algorithmic contribution therefore adds little over a simple heuristic.

Overall, my third question is answered, but my concerns for Q1 and Q2 remain.

AuthorsOpenReview ↗

Q1–Q2: Worst-Case Safety Beyond Calibration and Conditional Value of Risk Allocation

We thank the reviewer for the follow-up on safety beyond calibration (Q1) and the predictive value of candidate fidelity (Q2). We distinguish certified safety from candidate-dependent predictive gains.

Q1. Why a worst-case guarantee beyond decision-preserving calibration?

We agree that accuracy-preserving calibration is a relevant comparator. Consider human-in-the-loop screening: an approved top-1 action remains fixed, while alternative-class probabilities inform expert follow-up. In such a setting, an edit can severely suppress a possible true class without changing the action. For example, p=(.70,.20,.10)p=(.70,.20,.10) and q=(.80,.199,.001)q=(.80,.199,.001) have identical rankings, yet incur log-loss regret log⁡100≈4.61\log 100\approx4.61 if class 3 occurs (Theorem 2). These settings motivate per-outcome safeguards; we do not claim deployment validation.

On the same 10,000 ImageNetV2 examples and anchor reference, per-example worst-label log-regret max⁡ylog⁡(py/ry)\max_y\log(p_y/r_y) is:

MethodChanged decisionsP95P99Max
Source-validated temperature scaling00.7901.2301.710
Decision-only projection (Table 5)01.0151.0611.096
Joint safe ray / PSPL (Table 5)00.1000.1000.100

This compares tail risk, not matched calibration or predictive quality. TS has lower P95 but higher P99 and maximum regret than decision-only projection. Neither certifies a uniform bound from decision preservation alone. Decision-only projection also increases NLL from 1.362 to 1.371. By contrast, with PSPL's local log cap 0.1, worst-label regret is at most 0.1 and every class retains at least 90.48% of its anchor probability. Theorems 1 and 3 give worst-case average conditional-loss protection for arbitrary row-specific label distributions. Under the ledger's twofold local cap 0.2, the per-row floor is 81.87%, with cohort budget 0.1. This certifies risk rather than dismissing calibration. Indeed, TS+PSPL improves NLL over TS on OfficeHome (0.9644 → 0.9293, Table 7) and VLCS (0.8115 → 0.7655, Table 9), without top-1 changes.

Q2. What predictive value does allocation actually establish?

Retaining more of a poor candidate need not improve prediction; the ledger does not identify beneficial edits. Its objective has a conditional proper-score interpretation. Let ηi\eta_i be the unknown conditional label distribution, qiq_i the candidate, rir_i the allocated edit, D=∑i∥ri−qi∥22D=\sum_i\|r_i-q_i\|_2^2, and E=∑i∥qi−ηi∥22E=\sum_i\|q_i-\eta_i\|_2^2. For normalized expected Brier risk,

=\frac{D+2\sum_i\langle r_i-q_i,q_i-\eta_i\rangle}{nK} \leq\frac{D+2\sqrt{DE}}{nK}.$$ If $q_i=\eta_i$, the excess expected Brier risk is exactly $D/(nK)$. With imperfect candidates, a smaller $D$ tightens this upper bound for fixed candidate error $E$, but **does not prove that ledger beats uniform allocation on realized NLL or Brier**. The anchor-relative constraints separately limit worst-case deterioration; they do not make an inaccurate candidate accurate. To examine this limitation, we stratified *frozen* ImageNetV2 allocations **after** observing labels, by candidate-versus-anchor NLL and Brier changes: | Candidate vs. anchor | Images | Ledger − uniform NLL | Ledger − uniform Brier ($\times10^{-6}$) | |---|---:|---:|---:| | Both scores improve | 3,400 | −0.020000 | −4.000 | | Mixed changes | 2,100 | −0.003000 | −0.800 | | Both scores worsen | 4,500 | +0.010627 | +2.413 | | Whole cohort (Table 3) | 10,000 | −0.002648 | −0.442 | The ledger helps on the first two groups but hurts on the 4,500 examples where the candidate worsens both scores. Group-weighted differences match the cohort result to rounding. This is a retrospective diagnostic of candidate-dependent effects, **not** evidence that the ledger can prospectively identify good candidates. Under identical twofold local caps and budgets (Table 3), ledger reduces squared candidate distance from **175.78 to 123.39** on ImageNetV2 and **211.26 to 196.10** on VLCS. The corresponding NLL changes are modest: **1.343168 → 1.340520** and **0.787419 → 0.785049** (uniform equal-cap fraction → ledger). We do not interpret the original VLCS intervals as establishing a nonzero mean NLL improvement. Thus, the defensible allocation claim is **greater candidate fidelity under specified risk constraints**, not guaranteed target-domain predictive improvement. Predictive benefits remain contingent on candidate quality. We appreciate the reviewer’s careful scrutiny of these distinct claims.

Thank you for the direct answers.

Q1. The tail-risk table makes the difference from calibration methods clear. But the value 0.100 for PSPL is the cap itself, and the screening scenario is not validated, as you say. So I still do not see evidence that this guarantee matters in practice.

Q2. I appreciate the clear statement that the claim is candidate fidelity and not predictive improvement. The stratified table agrees with this: the ledger is worse than uniform allocation on the 4,500 images where the candidate is worse. In Table 3, the simpler distance-priority rule also has slightly lower Brier and NLL than the ledger on ImageNetV2.

Overall, the responses clarified the claims, and I thank the authors for that. But the clarified claim is narrower than the one in the submission, and my concern about its significance remains. I therefore keep my score.

Reviewer_tKAPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper analyze the risk in editing probability vector of a model.

Critical strengths

None

Critical weaknesses

Weaknesses

  1. Poorly structured and inadequately motivated introduction: The introduction is not well written and fails to establish a clear background and motivation for the research problem. In particular, the authors do not adequately explain the application domains or specific tasks addressed by the proposed method, especially those involving probability vectors as model outputs. Consequently, the reader is not provided with sufficient context to understand the significance of the research question, the limitations of existing approaches, or the necessity of the proposed framework.

  2. Inconsistent and insufficiently defined terminology: The paper introduces numerous technical terms without providing clear definitions or maintaining consistent terminology. For example, the first paragraph of the introduction uses the terms "probability edits" and "altering probability values" without clarifying whether they refer to the same concept or different operations. Similarly, the second paragraph introduces terms such as "complete ranking preservation" and "log loss deterioration" without sufficient explanation or contextualization. These inconsistencies and ambiguities significantly affect the readability of the manuscript and make it difficult to understand the underlying problem formulation.

  3. Lack of logical progression and clarity in the introduction: The third paragraph begins with a seemingly subjective statement that is not supported by appropriate evidence or citations. Furthermore, the authors introduce the term "first row" without first explaining the corresponding matrix, dataset, or experimental context. The reader is then directed to Figure 1, where additional terms such as "anchor" and "anchor action" are introduced without adequate definitions. This abrupt presentation of concepts, without a coherent explanation of their relationships, makes the introduction difficult to follow. More importantly, the paragraph outlining the paper's contributions is particularly unclear, leaving this reviewer unable to identify the principal methodological contributions or their novelty relative to existing approaches.

  4. Insufficient methodological clarity: The methodology section suffers from similar issues regarding terminology and conceptual explanations. Key concepts such as "anchor distribution," "black-box candidate," and "safe edits" are introduced without sufficiently precise definitions or intuitive explanations. As these concepts appear central to the proposed framework, their lack of clarity makes it difficult to understand the methodology, its underlying assumptions, and how the different components interact.

  5. Inadequate description of the experimental design and evaluation protocol: The manuscript does not provide a sufficiently clear discussion of how the experiments are designed and conducted. In particular, the experimental settings, evaluation metrics, and their relevance to the research objectives are not adequately explained. Without a clearly defined evaluation protocol, it is difficult to assess whether the reported experiments appropriately validate the proposed method and support the authors' claims.

Overall, the primary weakness of the manuscript is its lack of clarity in presenting the research problem, methodological formulation, and experimental evaluation. The frequent introduction of undefined or ambiguous terminology, combined with the absence of a coherent narrative, makes it challenging to evaluate the technical contributions and scientific validity of the proposed approach. Substantial revisions to the introduction, methodology, and experimental description are necessary to make the work accessible and its contributions assessable.

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Clarifying the Motivation, Method, and Evaluation of Risk-Budgeted Probability Editing

Thank you for identifying the presentation and motivation issues. We agree that the deployment setting and evaluation should be clear before the optimization details. We have clarified the Introduction, added quantitative comparisons in Appendix C, and qualified the claims in the Abstract and Conclusion. All references below use the revised manuscript's numbering.

1. Setting and terminology. Consider an image classifier on an unlabeled target cohort. For input xix_i, the deployed predictor gives anchor probabilities pip_i, while a separately selected adaptation, calibration, or pooling procedure proposes candidate probabilities qiq_i. Our returned edit is rir_i. A row is one input and its probability vector. The anchor action is the decision selected from pip_i, such as its top-1 class. Probability editing changes the vector without necessarily changing that action. The revised Introduction defines the input, anchor, candidate, and returned edit. It positions the method as a safety layer applied after predictor selection. It cannot correct an incorrect locked action, and unrestricted adaptation may be preferable when action changes are acceptable.

2. Why action preservation is insufficient. For p=(0.70,0.20,0.10)p=(0.70,0.20,0.10) and q=(0.80,0.199,0.001)q=(0.80,0.199,0.001), the strict ranking is unchanged, but label 3 incurs log regret log⁡(0.10/0.001)=log⁡100≈4.61\log(0.10/0.001)=\log 100\approx4.61. This example now appears in the Introduction and Section 3.1. A pointwise log cap of 0.1 requires ry≥e−0.1py≈0.9048pyr_y\ge e^{-0.1}p_y\approx 0.9048p_y for every class. In the ledger experiments, the twofold local log cap is 0.2 and the cohort budget for mean worst-label log regret is 0.1. Section 3.2 defines these constraints, and Section 4.2 reports their values.

3. Guarantee and contribution. Theorem 1 identifies worst-label regret ρi=max⁡y[ℓ(ri,y)−ℓ(pi,y)]\rho_i=\max_y[\ell(r_i,y)-\ell(p_i,y)] with the maximum conditional expected loss increase over all label distributions for that input. Theorem 3 extends this to the cohort: the average of these regrets equals the worst-case average conditional regret with arbitrary row-specific label distributions. The simplex identity is elementary. Its role is to make risk allocatable: local action and Brier/log caps constrain each row, while two shared budgets control the cohort. This deterministic guarantee bounds loss relative to the anchor on the fixed cohort in hand. Future accuracy and calibration are outside its scope. The revised Abstract and Discussion make this boundary explicit.

4. Algorithm and objective. Figure 1 and Section 3.2 describe the sequence. Proposition 1 gives the convex decomposition. The Pathwise-Safe Prediction Layer (PSPL) computes each maximum locally admissible step γˉi\bar\gamma_i. The ledger chooses ri=pi+γi(qi−pi)r_i=p_i+\gamma_i(q_i-p_i), with 0≤γi≤γˉi0\le\gamma_i\le\bar\gamma_i, to minimize ∑i∥qi−pi∥22(1−γi)2\sum_i\|q_i-p_i\|_2^2(1-\gamma_i)^2 under the two cohort budgets. The optimizer uses two shared dual prices for the cohort budgets, then solves a scalar problem within each row's local bounds. The edit is chosen for closeness to the candidate because target scores are available only after labels arrive. Exact safe bodies and sparse hulls support this construction.

5. Evaluation and interpretation. ImageNetV2 uses 10,000 frozen predictions, K=1000K=1000, ϵL=.1\epsilon_L=.1, and ϵB=.05/K\epsilon_B=.05/K. VLCS uses four domains and five fixed seeds, with ϵL=.1\epsilon_L=.1 and ϵB=.005\epsilon_B=.005. Labels enter evaluation only. We report decision changes and worst-label regret for safety, distance and retained update for fidelity, and Brier and negative log-likelihood (NLL) for predictive quality.

Section 4.2 distinguishes the comparisons. With matched cohort budgets and different local caps, the ledger raises retained update over pointwise clipping from 19.9% to 30.0% on ImageNetV2 and from 14.9% to 22.2% on VLCS. Under identical twofold caps and budgets, the ledger raises retained update over equal-cap-fraction allocation from 19.962% to 29.955% on ImageNetV2 and from 17.458% to 22.174% on VLCS. The relative gains are therefore 50.06% and 27.01%, respectively, against that comparator.

Against the stronger marginal-efficiency heuristic, the squared-distance gain is much smaller: 123.39 versus 123.91 on ImageNetV2, and a paired ledger-minus-heuristic difference of -0.346 (95% CI [-0.532,-0.161]) on VLCS. Appendix C and Table 3 report matched-cap predictive scores and measured runtimes. Section 4.2 explains the computational trade-off. Section 4.1 reports no action changes and no safety-budget violations above 10−1010^{-10} in the million-pair audit.

We appreciate the reviewer's suggestion to distinguish the guarantee from the evaluation. The predictive-score comparisons and timing protocol in Appendix C are additions from this rebuttal stage. The original supplementary archive provides the core method implementations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.