- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies label-free probability editing relative to an anchor predictor. It formulates safety through worst-label Brier and log-loss regret, combining per-example caps with shared cohort budgets while preserving the anchor’s decisions. A convex allocator minimizes squared distance to externally proposed candidate predictions and decomposes through two dual risk prices.
ImageNetV2 and VLCS experiments demonstrate increased candidate-update retention, supplemented by predictive-quality evaluations and large-scale numerical safety audits.
Critical strengths
- A precise and operational safety formulation.
The method bounds degradation for every possible label while preserving the deployed action. Combining local caps with a cohort budget provides a clear way to permit heterogeneous edits without allowing unrestricted harm on individual examples. This is useful when target labels are unavailable and the proposed updates may be unreliable.
- Tractable optimization backed by implementation audits.
The two-price decomposition, analytic ray endpoints, and certified sparse approximations make the formulation computationally accessible. The reported stress tests and explicit treatment of numerical precision and decision ties strengthen confidence that the mathematical constraints can be enforced in practice.
Critical weaknesses
- The headline improvement may overstates what the comparisons establish.
The headline improvement uses looser per-example caps, while gains over a strong heuristic under matched constraints are modest. The exact allocator’s added value therefore remains insufficiently demonstrated.
Questions for authors
-
Which comparison supports the conclusion’s approximately 50% improvement under “matched budgets and caps”? What is the retained-update improvement over the strongest heuristic when both cohort budgets and local caps are identical?
-
Under those identical constraints, how do the ledger and marginal-efficiency heuristic compare in realized Brier/NLL and computational cost?
Separating Budget-Matched and Cap-Matched Gains and Quantifying Utility and Cost
Thank you for pointing out the ambiguity in the headline comparison. We agree that the comparator, local caps, cohort budgets, and metric must be explicit. We have revised the Abstract, Section 4.2, and Conclusion. The new Table 3 in Appendix C reports matched-cap results and measured runtimes, with paired predictive-score contrasts reported below the table. References below follow the revised manuscript's numbering.
Q1. What supports the approximately 50% figure? Figure 2a holds the cohort budgets fixed. Pointwise clipping uses onefold row caps and the ledger permits twofold caps. Retained update rises from 19.9% to 30.0% on ImageNetV2 (approximately 50.7% relative) and from 14.9% to 22.2% on VLCS (+7.30 points; 95% CI [5.94,8.66]). This is a budget-matched comparison.
Our additional comparison matches both twofold caps and cohort budgets. Compared with equal-cap-fraction allocation, the ledger raises retained update from 19.962% to 29.955% on ImageNetV2 (+9.993 points, +50.06% relative). The corresponding VLCS comparison is 17.458% to 22.174% (+4.716 points, +27.01% relative). The approximately 50% gain at matched caps refers specifically to the ImageNetV2 comparison with equal-cap-fraction allocation. The revised Abstract reports these cap-matched endpoints; the Conclusion gives the dataset-specific relative gains.
Q2. Improvement over the strongest heuristic. Figure 2b-c and Table 3 compare five rules under identical twofold caps and budgets, all with zero action changes. ImageNetV2 total squared candidate distance is 123.39 for the ledger, 123.91 for marginal efficiency, 123.99 for distance priority, 175.78 for equal-cap-fraction allocation, and 191.32 for confidence priority. The ledger improves on marginal efficiency by 0.52 (0.42%). On VLCS, the paired ledger-minus-marginal difference is -0.346 (95% CI [-0.532,-0.161]) across 20 domain-seed conditions.
Retained Euclidean update is a different metric. Marginal efficiency retains 30.18% versus 29.955% for the ledger on ImageNetV2, about 0.23 points more, and approximately 0.11 points more on VLCS. The global optimality claim applies to the specified squared-distance objective, up to numerical tolerance.
Q3. Predictive utility under matched constraints. Table 3 reports the following measured scores:
VLCS values are means over 20 conditions. ImageNetV2 ledger-minus-marginal differences are −5.13×10−7 in Brier and -.001954 in NLL. VLCS paired differences, computed before averaging, are −4.61×10−5 in Brier (95% CI [−6.86,−2.37]×10−5) and -.001932 in NLL ([-.003198,-.000666]). Both VLCS intervals exclude zero across the 20 paired conditions. Those conditions share four domains, so the intervals describe the observed cohorts. On ImageNetV2, distance priority attains slightly lower Brier and NLL than the ledger. Candidate fidelity and predictive quality therefore require separate evaluation. Appendix C reports the paired contrasts.
Q4. Additional computation. Marginal efficiency uses a priority rule. The ledger optimizes two cohort-level dual prices and solves scalar row minimizers by vectorized bisection at each evaluation. Section 3.2 specifies O(nK) preprocessing and O(MnT) allocation for M dual evaluations and T bisection steps, followed by numerical feasibility repair.
Both methods used the same arrays, caps, budgets, tolerances, and eight CPU threads on an AMD EPYC 7543 (32 physical cores, 64 logical threads) with GPU disabled. The environment was Ubuntu 22.04.5 LTS, Python 3.9.19, NumPy 1.26.4, and SciPy 1.13.1. We used one untimed warmup and ten timed runs per cohort with
time.perf_counter. The ± values are sample SDs (ddof=1) across ten ImageNetV2 runs or 20 VLCS cohort means, each averaging ten runs. Timing includes local-cap preprocessing, allocation, bisection where applicable, and feasibility checks and repair. It excludes inference, prediction generation, disk I/O, and loading.Section 4.2 and Appendix C now document this trade-off. Marginal efficiency may be preferable when latency matters most. The ledger offers the minimum squared distance at additional measured cost. The ratios apply to these implementations and cohorts. We appreciate the reviewer's request to quantify this trade-off.