acceptodds
Under review as a conference paper at ICLR 2027

ACTIVE LEARNING WITH UTILITY-TAILORED STRICTLY PROPER SCORES

Abstract

Medical active learning must value labels according to the decisions that predictions support, while preserving accurate probability estimation. We introduce Domain-Utility-Sensitive Active Learning (DUSAL), which constructs a strictly proper score from a complete action–utility table. DUSAL-Hard adds normalized Bayes value to a strictly proper base potential. Its divergence decomposes exactly into predictive divergence and normalized decision regret; we characterize this augmentation up to affine terms. Under exact Bayesian updating, acquisition separates into Bayesian Estimate of Mean Proper Scores (BEMPS) and clinical expected value of sample information (EVSI). We also derive representation invariances, a smooth utility-gap variant, and predictive and decision convergence under explicit progress assumptions. Exact finite-model experiments demonstrate decision-regret reductions through boundary-sensitive acquisition and within-region smoothing, alongside prediction–decision trade-offs. Across five public medical tasks, DUSAL makes label selection sensitive to utility information, producing measurable differences from matched BEMPS. Although additional cost gains from utility augmentation remain unconfirmed under Holm correction, DUSAL-Div achieves Holm-controlled reductions in realized-cost area under the learning curve (AULC) on three of four tasks relative to top-bselection. Comparable gains with BEMPS-Div demonstrate the practical benefit of batch diversification.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Active learning, strictly proper scoring rules

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V8TaOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The manuscript introduces two new decision-aware acquisitions rules in active learning of KK-valued nominal outcome by augmenting Savage score pertaining to strictly convex function with additional term, which is either normalised Bayes value (DUSAL-Hard) or its soft version (DUSAL-Soft). Bayes value corresponds to normalised value of maximal expected utility. Theorem 1 shows that the resulting divergence can be decomposed into sum of the regret pertaining to the original loss and the selected action’s scaled decision regret. Such decomposition can be essentially obtained only by the above augmentation construction (Theorem 2). The introduced utility adjusted scores are than used to construct acquisition strategy based on maximisation of the gain resulting from querying candidate vector of predictors xx.

Critical strengths

Nice incorporation of Bayes-value augmentation term pertaining to utilities into construction of new proper scores which can be used in active learning.

Critical weaknesses

The exposition is laconic at places and its essence will be hard to grasp for readers not well acquinted with the field of active learning.

Questions for authors
  1. How the performance of the studied acquisition strategy depends on deployment distribution μn\mu_n and weights (ωi)(\omega_i) ?
  2. The scaling of embeddings in Algorithm 3 using (ωm)\sqrt(\omega_m) is not clear.
2 comments

I am satisfied with Authors' answers to my questions. As I wrote in my appraisal, the idea behind DUSAL-Hard is original and worth pursuing. I keep my score.

Dear Reviewer V8Ta,

Thank you for confirming that our responses addressed your questions. We appreciate your constructive feedback and your recognition of DUSAL-Hard's originality and research potential. Thank you for your thoughtful engagement with our work.

Reviewer_8LRhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Domain-Utility-Sensitive Active Learning (DUSAL), which constructs a strictly proper score from a complete action–utility table. DUSAL-Hard adds normalized Bayes value to a base potential, giving a divergence decomposition into predictive divergence and normalized decision regret. Under exact Bayesian updating, acquisition separates into BEMPS and clinical EVSI. The paper also provides a smooth variant, DUSAL-Soft, a batch-diversification version, DUSAL-Div.

Critical strengths

The motivation is interesting, especially for medical active learning, where a label’s predictive value may differ from its value for a clinical decision. Using a complete action–utility table makes the consequences of different actions explicit. This is a meaningful problem to study when expert labels are limited and different errors have different costs.

Critical weaknesses
  1. The acquisition decomposition relies on exact Bayesian updating. In practice, the method reweights a finite ensemble for hypothetical updates, but retrains the ensemble after acquiring labels. The refit audits do not establish agreement between these procedures. It is unclear whether the theoretical decision-value interpretation carries over to the implemented algorithm.
  2. Hard and Soft show no additional cost gains over matched BEMPS after Holm correction in the confirmatory studies. Div improves performance, but BEMPS-Div gives comparable gains. These results mainly support batch diversification rather than the proposed utility-based addition.
    1. Simple entropy baseline outperforms BEMPS on all three tasks in the binary replication, and Hard does not consistently close this gap. The main studies also lack comparisons with established batch methods such as BADGE or BatchBALD. In particular, comparing Div only with top-b selection is insufficient to show that its gains go beyond existing diversity-based methods.
Questions for authors
  1. In Table 11, Hard changes realized-cost AULC relative to BEMPS by +0.12%, −0.04%, and −0.44% on BreastMNIST, PneumoniaMNIST, and ADE; Soft gives −0.19%, +0.03%, and −0.71%. None passes Holm correction. Given these results, what practical benefit of utility augmentation is established by the current experiments?
  2. In Table 18, Hard-Div reduces cost AULC relative to Hard top-b by 3.85%, 1.09%, and 1.16% on PathMNIST, PneumoniaMNIST, and ADE. BEMPS-Div also gives reductions of 3.10%, 1.27%, and 1.38% relative to BEMPS top-b. Do the direct Hard-Div versus BEMPS-Div comparisons show any additional benefit from utility augmentation?
  3. In Table 14, Hard has higher cost AULC than entropy sampling by 1.99%, 0.79%, and 1.73% on BreastMNIST, PneumoniaMNIST, and ADE, although not all differences are statistically resolved.
Reviewer_HJWKOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes DUSAL, an active-learning method that chooses which examples to label based on both prediction quality and the consequences of downstream decisions. DUSAL combines a standard probability-quality score with a utility table describing the consequences of each action under each possible outcome. The Hard variant values information that could change the optimal action. The Soft variant smooths this objective, allowing it to distinguish prediction improvements even when one additional label cannot change the action. A batch variant, DUSAL-Div, selects diverse examples according to their expected effects across a reference population. The main theoretical result is that the Hard variant’s score error separates into prediction error and the extra cost of choosing a suboptimal action, while still rewarding accurate probability estimates. Under exact Bayesian updating, the acquisition objective similarly separates into predictive learning value and the expected value of decision information. Synthetic experiments demonstrate that utility-aware selection can reduce decision regret. However, on five public medical tasks, DUSAL changes which examples are selected without establishing an improvement over the predictive-only BEMPS baseline.

Critical strengths
  • Addresses an important problem: The paper studies how active learning can account for downstream decision consequences, which is especially relevant in medical settings where prediction errors have unequal costs.

  • Provides a principled theoretical foundation: The method preserves strictly proper scoring while offering an intuitive decomposition of acquisition value into predictive learning and decision information, making the trade-off between probability accuracy and decision utility explicit.

Critical weaknesses
  • Clarity and presentation: The paper relies heavily on abstract terminology and dense mathematical notation, without enough intuitive explanation. This makes the central ideas and the connections between the theory, algorithm, and experiments difficult to follow.

  • Limited evidence of practical benefit: The experiments do not establish that utility tailoring improves performance over matched BEMPS on public datasets. The observed gains primarily come from batch diversification, which also benefits the baseline.

Questions for authors

Pleases see weakenesses.

Comments on the paper

5 comments
Area_Chair_ejP7OpenReview ↗

Regrading the paper scope

Dear authors,

Thanks for your submission. I quickly read the review and I may omit important details. So basically I have several clarification questions. These are confusions. If possible, can you explain them in plain language? Thank you.

  • Can you explain what are strictly proper scoring rules? It's the rule that specifies certain properties?

  • Medical active learning must value labels according to the decisions that predictions support, while preserving accurate probability estimation. I can roughly understand this context by the active learning scoring should hold certain useful properties. My question what is the actual implication in medical AI. For example, medical AI always want better models like transparent, fair or reliable. My question is not doubting the necessary but requiring a concrete context. For example, how a scoring system in active learning change the medical AI.

  • How the experiments justify the medical-AI value?

Dear Area Chair,

Thank you for these questions. DUSAL studies how limited annotation effort can support both accurate probability estimates and decisions that account for the consequences of errors.

1. Explanation of Strictly Proper Scoring Rules

A strictly proper scoring rule is a grading system that is designed to reward a forecaster, whether a human expert or an AI model, for honestly reporting their probabilities. Imagine grading a weather forecaster who looks at their data and genuinely believes there is a 70% chance of rain. They might be tempted to report a different probability:

  • They might say 100% to sound bold and confident.
  • They might say 50% to appear cautious.

A scoring rule is the mathematical formula used to award points based on the reported probability and the actual outcome: it rained, or it did not. The terms describe how that formula rewards predictions:

  • Proper: Reporting the forecaster's genuine belief is a strategy that achieves the highest expected score. Other reports may tie, but none scores better in expectation.
  • Strictly proper: Reporting that exact belief is the only way to achieve the highest expected score. Honesty is the unique best strategy for maximising expected points.

Here, "expected" means averaging the possible grades using the forecaster's belief: a 70% chance of rain and a 30% chance of no rain. Strictly proper scoring therefore rewards honest probability reporting. Gneiting and Raftery, 2007.

2. Medical-AI Implications

In medical-AI development, active learning helps decide which unlabelled scans or clinical notes should receive specialist annotation next. DUSAL makes that choice depend on both what a label could teach the model about probabilities and how that learning could improve decisions under a specified cost table.

Focusing Specialist Time on Decision-Relevant Information

A large improvement in a probability estimate does not always change the preferred action. BEMPS prioritises labels expected to improve probability predictions. DUSAL-Hard also considers whether that information could improve decisions under the cost table. In our paper's constructed screening example, a missed positive case carries 19 times the penalty of a false alarm. This gives a 5% threshold for a positive declaration within that research problem. To illustrate this mechanism, consider two hypothetical image annotations:

  • Image A could clarify predictions near the threshold, revealing that a different action is preferable.
  • Image B could produce a larger predictive improvement while leaving the affected decisions unchanged.

BEMPS selects B in the constructed example. DUSAL-Hard selects A because its additional decision value outweighs B's predictive advantage. Utility therefore can reverse a ranking, not merely break a tie (Appendix D.1, p. 23; Tables 5-6, p. 24). The value is assessed across reference cases: the image selected for annotation need not be the case whose eventual decision changes. Under exact Bayesian updating, Theorem 3 separates DUSAL-Hard's estimated value of obtaining a label into predictive improvement and expected decision benefit (Section 3.3, pp. 5-6). The complete table also considers several possible outcomes leading to fewer actions. The BloodMNIST and PathMNIST studies exercise this setting by mapping eight and nine classes, respectively, to two research-defined actions, with explicit penalties for their consequences (Section 4.2, p. 8).

Preserving the Meaning of Risk Estimates

A probability estimate and a decision threshold serve different purposes. The former describes the chance of an outcome; the later reflects the consequences of the available actions. In the 5% threshold problem, a 10% estimate already favours the positive declaration. Some cost-weighting shortcuts can blur this distinction. Specifically, directly weighting a logarithmic prediction loss by unequal class costs can make distorted probabilities score best. DUSAL instead incorporates decision consequences while preserving the true probabilities as the unique expected-score optimum for both Hard and its smooth variant, DUSAL-Soft (Theorem 1, p. 4; Section 3.1 and Proposition 1, p. 5).

E5 quantifies the value and cost of retaining predictive learning: DUSAL-Hard improves probability accuracy relative to utility-only selection, but incurs higher budget-averaged decision regret. This makes the trade-off visible, rather than treating accurate probabilities and minimum decision cost as interchangeable goals (Table 7, p. 25). DUSAL makes the purpose of annotation explicit: improve probability estimates while valuing the decisions they support.

3. Experimental Justification

Our experiments demonstrate how decision consequences can guide label selection, identify practical benefits on medical benchmarks, and provide checks for interpreting those benefits. We use research-specified cost tables to make the consequences being evaluated explicit.

Controlled Tests Demonstrate Decision-Aware Learning

We first use the constructed problems in which the effects of possible labels can be calculated exactly. These tests isolate the selection mechanism from neural-network approximation errors and demonstrate four capabilities (Appendices D.1-D.2; Tables 5-7, pp. 23-25).

  • Directing labels toward consequential decisions (E1 and E2). In the screening construction, DUSAL_Hard uses all 50 requests on decision-relevant information; BEMPS uses none for that purpose. DUSAL_Hard eliminates decision regret at the end of labelling, meaning the extra expected cost above the best action given the true probabilities. E2 tests a three-action problem: when requests offer equal predictive improvement, DUSAL_Hard distinguishes them by the consequences of the decisions they could change.
  • Recognizing useful learning before an action changes (E4). DUSAL_Soft responds to changes in the relative attractiveness of actions even when one label cannot immediately change the preferred action. In the associated simulation over 100 paired runs, mean accumulated decision regret falls from 1.870 to 1.446, a 22.7% reduction relative to BEMPS when its near-ties favour decision-irrelevant requests. Reductions remain positive across all tested tie-handling rules, ranging from 14.3% to 24.4%.
  • Using the consequences of each action (E3/P3). P3, the randomised extension of E3, changes individual action penalties while keeping their total for each outcome fixed. DUSAL_Hard and DUSAL_Soft adapt their selected example in all 100 paired problems; BEMPS and a comparator that uses those totals to weight prediction errors switch in none. This demonstrates what the complete action-utility table adds beyond a simpler cost summary.
  • Retaining probability quality alongside decision value (E5). Across 100 paired simulated problems, DUSAL_Hard lowers budget-averaged Brier score by 9.09% and negative log-likelihood by 6.02% relative to utility-only selection. These measures assess probability-prediction quality. Budget-averaged decision regret increases by 25.06% under an equal mixture of three evaluation cost ratios. Reporting both outcomes makes the value and cost of retaining predictive learning explicit.

Medical Benchmarks Identify Measured Benefits

Five public tasks cover BreastMNIST, PneumoniaMNIST, adverse drug event (ADE) text classification, BloodMNIST, and PathMNIST. Within each study, methods share labelling budgets and training conditions. The main measure averages penalties on held-out decisions across the labelling process, reported as realised-cost area under the learning curve (AULC) (Section 4.2 and Table 1, p. 8).

Batch diversification improves this measure on three of four tested tasks. Div selects representative examples according to their predicted learning effects across reference cases. Each reduction below compares Div with selecting the highest-scoring batch under the same acquisition score:

TaskHard-Div Cost ReductionBEMPS-Div Cost Reduction
PathMNIST3.85%3.10%
PneumoniaMNIST1.09%1.27%
ADE1.16%1.38%

All six reductions survive Holm correction for multiple comparisons within their respective four-task families. The matched results support diversification as the demonstrated source of these gains (Section 4.3 and Table 2, p. 9).

Utility also changes annotation priorities. In binary validation diagnostics, 19%-29% of DUSAL's selected items are absent from matched BEMPS sets. Additional utility-specific cost gains remain unconfirmed under Holm correction in the evaluated confirmatory comparisons. Exploratory results and conditions excluded during preliminary screening are kept separate from those conclusions. Thus, the studies distinguish sensitivity to decision consequences from demonstrated cost improvement (Sections 4.2-4.4, pp. 8-10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.