acceptodds
Under review as a conference paper at ICLR 2027

On Reward Design for Aligned Decision-Making under Uncertainty

Abstract

In human-AI cooperation, a utility function elicited from a human decision maker is routinely repurposed as a reward for the artificial agent. When the agent communicates set-valued predictions, extending the original utility to a reward over imprecise information is nontrivial, as the expanded utility has many more degrees of freedom than the original. Existing methods assume that the value of an imprecise piece of information reduces to an aggregate over its constituent precise states, an assumption that is frequently violated. We propose a general framework that makes no such reduction. Decider's behavior under uncertainty is explicitly modeled via selection strategies induced by decision criteria. We introduce the notion of validity, which ensures that a criterion respects the rationality encoded in the utility function, and the stronger notion of alignment, which guarantees that the resulting reward faithfully preserves the original preferences. We prove constructive compatibility results between classical criteria (Laplace, maximin, maximax, minimax regret) and standard aggregators (min, max, avg), and provide an algorithm to build aligned rewards. Our framework accommodates arbitrary sets of terminal acts, bridges the gap between utility elicitation and reward design for set-valued information, and generalizes naturally to richer uncertainty representations.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
reward design, utility elicitation, reward alignment, human–AI cooperation, decision-making under uncertainty, set-valued predictions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Sun3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a decision model, where the DM can consult an information assistant (IA) for (epistemically interpreted) information about the states before making their final decision. Concretely, the IA proposes a set F of states interpreted as "the true state of the system is within F" and the DM can pick their final action based on this info. Compared to earlier similar work, the paper proposes two generalizations:

  1. The DM can potentially also pick non-singleton terminal acts.
  2. The DM does not have to pick from F, even if the IA provided this information.

The paper argues that other models (not allowing for 1. and 2.) face severe practical limitations and therefore need to be adapted. The paper does so by proposing ways to construct specific selections rules that overcome these limitations. Concretely, it is shown how a reward function for the IA can be constructed that is aligned well-enough with the DM's utility function and a specific criterion such that the limitations models of traditional models can be avoided. The paper ends with a simulation study illustrating that the problem is actually relevant in certain scenarios (under different of random utility functions of the DM).

Critical strengths
  • The paper is clearly written and explains the limitations of exisiting apporaches well. The toy examples help to make the paper very accessible to a broad audience. The problem of aligning IA better with an human DM seems to be very relevant.

-The decision model is mathematically rigorous and specifically addresses the limitations identified. The proposed construction of a reward function based on the DM's preferences and a general decision criterion is elegant and the characterizations given in Theorems 8.1-8.4 are as far as I can see correct.

Critical weaknesses

The paper didn't manage to fully convince me of the practical relevance of the concrete problem studied. The only motivation given for this is via toy examples that are SPECIFICALLY CONSTRUCTED to demonstrate the limitations of the existing models. This alone is not a convincing argument or justification of the proposed generalization. While the simulation study did a small steop in this direction I really feel the paper would benefit a lot from a proper application example.

Questions for authors
  • Is it possible to add more motivation why the problem is actually relevant or even a (small) application example?
2 comments
  • We thank the reviewer for their comments. We tried to address the concerns regarding motivation and problem relevance. We improved our experimental setting and hope to have addressed them. As these were common concerns across the reviewers, we wrote a common response. We would be glad to answer further questions and welcome any additional feedback.
Reviewer_Sun3OpenReview ↗

Thanks

I thank the authors for taking the time to answer. I keep my positive score.

Reviewer_SiQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a reward design framework that stays coherent according to a human-based decision maker. An AI assistant sends a set of information (states) to the decision maker, and decision maker makes some decisions that may contain uncertainties sometimes because of both the ai assistant or the decision maker. The designed reward function stays coherent with the global known utility function following the choices of decision maker on the set of actions.
The assumption is that the utility function on (state, action)s is known in advance. Another assumption is that the deciders make decisions based on the received information from the AI assistant, which are not always exact due to the available information/states. For this reason choosing the correct action is not always about selecting the action with maximum utility value, but is about choosing a most robust action using various approaches (e.g. expected utility, maximin, minimax regret). Following this motivation, the paper proposes a reward design framework with theoretical grantees that stays coherent w.r.t the choices of the decision maker.

Critical strengths
  • The strongest part of this paper is its theoretical guarantees and proofs on the proposed reward function and the designed framework.
  • The paper has selected an interesting question to study. The problem of uncertain environments due to partial provided information or uncertain decision makers is a common question in the decision-making community.
Critical weaknesses
  • The paper positioning w.r.t. the available literature requires some improvement. The paper first mention a broader version of related works in decision making, and closed loop interactions. However this work is narrowed down to one step reward design for a given set of states and selected set of actions by decision maker. Since the utility function and the decision-criterion of the decision makers are known in advance, the concern is about the novelty of this paper in constructing the reward function and its theoretical guarantees. The aggregation methods (defined in section 8) seems pretty straightforward. It'll be helpful to be specified by the authors which part of the proposed framework and its theory are new and which one comes from the available works in decision theory.
  • From practical and motivation point of view, the paper needs some practical examples or even some experimental results to better motivate it for the machine learning applications. For instance, is it possible to evaluate the proposed method on more realistic dataset such as selective classification, medical decision support or human-ai collaborative tasks. The paper validates the proposed framework using synthetic data, but it stays limited for larger and more realistic dataset and use-cases.
Questions for authors

Below there are some suggestions and less questions:

• The paper already presents in section 10, the complexity of the method, the wondering is that how this complexity affects the applicability of this method on larger problems? The authors can better discuss the computational bottlenecks and study is there a particular structure or formulation which can reduce this complexity. • The practical examples are difficult to follow conceptually, their improvement can help to better understand the main objective and main motivation of this work. For instance a concept like {C_1, C_2} refers to a general treatment for cardiac disease no1 and cardiac disease no2. However this notation means using a treatment as union of treatment for cardiac disease C_1 and cardiac disease C_2, but the paper introduced it as a general treatment for any cardiac disease. There are several similar examples in the paper currently.

1 comment
  • "From practical and motivation point of view, the paper needs some practical examples or even some experimental results to better motivate it for the machine learning applications.": We thank the reviewer for their comments. We approached the concerns regarding the handling of non-singleton terminal acts and the completion of medical example in expanding the experimental section. As those are common concerns, we better described the changes in the general response.
  • "It'll be helpful to be specified by the authors which part of the proposed framework and its theory are new and which one comes from the available works in decision theory.": Classical decision theory and imprecise probability frameworks provide the foundations: classical decision criteria (Laplace, maximin, etc.), standard aggregators (min⁡\min, max⁡\max, avg\mathrm{avg}), initial utility elicitation, and set-valued prediction. Our contribution is to combine and extend these tools to address reward design for set-valued predictions. We formalize selection strategies to explicitly capture the DM's behavior under uncertainty, without assuming that the value of imprecise information reduces to an aggregate of the values of its constituent precise states. We introduce validity (whether the criterion respects the supplied terminal utilities) and alignment (preserving the DM's preferences when extending those utilities). We establish the first constructive compatibility results between classical criteria and standard aggregators for this alignment requirement, together with an explicit algorithm to construct aligned rewards. We tried to clarify this in the paper.
  • "The paper already presents in section 10, the complexity of the method, the wondering is that how this complexity affects the applicability of this method on larger problems? The authors can better discuss the computational bottlenecks and study is there a particular structure or formulation which can reduce this complexity.": The main bottleneck for Algorithm 1 is the validity check of the given utility, as it requires iterating over every terminal act, which becomes heavy as the number of terminal acts grows. As we mention when computing the cost in the conclusion, this can in the limit grow exponentially. This will, however, seldom be the case. The more common usage will probably be setting the terminal acts to be something like almost all singletons and a small number of extra sets. In these scenarios, as we also showed, the number of terminal acts is much lower and the complexity is greatly reduced. In our complexity estimates we consider that the validity of the matrix is checked independently for each criterion/aggregator. This is not necessarily the case for an efficient implementation. For example, if the matrix contains only singletons, checking validity for one criterion implies validity for every one. Moreover, checking whether a matrix keeps validity when adding a new act is less expensive than computing everything from scratch. A particular structure or formulation that reduces the complexity is therefore available in the most common cases. We believe that imposing extra structure to uu can further simplify the problem. A final related concern is that, as the number of epistemic acts increases, the computation of the best answer by the IA will also increase. Our intuition is that computing the reward itself will not be the greater bottleneck of a proper implementation. Training a system with such a large number of possible acts might represent a bigger computational burden in practice.
  • "For instance a concept like {C_1, C_2} refers to a general treatment for cardiac disease no1 and cardiac disease no2. However this notation means using a treatment as union of treatment for cardiac disease C_1 and cardiac disease C_2, but the paper introduced it as a general treatment for any cardiac disease.": Indeed, the notation allows the set AA to mean two different things. We try to make them explicit in Table 1. On the one hand, the epistemic act of the IA represented by a set AA corresponds to its disjunctive interpretation: the IA answers "the actual state of the world belongs to AA, but there is not enough information to specify it further". On the other hand, the terminal act represented by AA is an action that the DM informed to always take if their knowledge about the state of the world corresponds to the epistemic act AA.
  • We hope having satisfactorily address all the points raised. We would welcome further questions or feedback, especially if any clarification is still needed.
Reviewer_4CR4OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

An assistant tells a decision maker "the world is in one of these states" (a set F), the decision maker then chooses an action, and the paper asks what reward the assistant should receive for having said F. The usual answer is to take the average, the min or the max of the decision maker's utility over the states in F. The paper's point is that this ignores what the decision maker can actually do with F: when a hedging action exists, "it is A or B" can be worth much more than the average of "it is A" and "it is B" (see Example 3.1). The reward is therefore defined by simulating the decision maker: given F, they apply a decision rule (Laplace, maximin, maximax, minimax regret) to select one or more actions, and the reward is the aggregated utility, in the true state, of the actions so selected. Two properties are required: validity (if the information received is itself an available action, e.g. "it is C1 or C2", the decision rule keeps it) and alignment (the new reward preserves the preferences of the original utility, which guarantees that no available action is discarded by the rule when it is exactly the information received, Theorem 7.2). The main results are four aligned rule/aggregator pairings (Theorems 8.1 to 8.4) and an algorithm that builds the reward once the decision maker's rule has been identified. The validation is a simulation on 100 random utilities with at most 5 states and singleton actions only: naive aggregation leaves up to 91.9% of information sets disposable at 5 states (Fig. 2), the proposed construction none, and a misidentified rule brings back up to about 40% (Appendix C.2).

Critical strengths
  1. A clearly identified flaw in existing aggregation-based rewards (Example 3.1): two epistemic acts receive the same reward although one of them allows a safe hedging act, so that the value of imprecise information depends on the structure of the utility function u and not only on the states in F. The framework handles an arbitrary set of terminal acts A ⊆ F, which according to the authors no prior work does, and the definitions of validity and alignment are clean and checkable.

  2. The link to AI alignment is concrete: a reward containing a disposable act gives the IA an incentive to withhold or distort information the decision maker would act on (the "paternalism" argument of Section 10), which is a concrete and simple misalignment mechanism.

Critical weaknesses
  1. The claimed advantage over prior work is, I think, not exercised, and nothing is learned. The first stated contribution is to handle arbitrary terminal acts A ⊆ F, which according to the authors no prior method does. The only experiment (Section 9, presented as illustrative) uses 100 random utilities, |Ω| ≤ 5 and A restricted to singletons, which is exactly the setting already covered by prior methods. The experiment establishes that disposable acts are frequent under naive aggregation, which is useful, but the existence of the phenomenon is already shown by Example 6.2, and no learning task, no agent trained with the reward and no real decision problem are presented, so that the "reward design for AI" framing is not really put to the test. The paper seems to me to belong to decision theory rather than machine learning, and its fit with ICLR is unclear.

  2. The constructive results cover four criterion/aggregator pairings (Theorems 8.1 to 8.4: avg, min, max and reg). The non-dominated criterion nd, which Section 6 presents as "particularly natural" and Example 6.2 uses, has no compatibility result: the paper therefore provides no aligned reward for this criterion. In addition, Algorithm 1 asks the decision maker to identify their own criterion; the authors themselves write that this "may be impractical" (Appendix C.2) and measure that a wrong identification brings back up to about 40% disposable acts at |Ω| = 5. The method thus depends on an input the paper acknowledges is hard to obtain, and errors in it are not benign.

  3. Section 10 (conclusion and perspectives) states that aligned rewards remove the IA's incentive to "deceive". The only result supporting this statement is Theorem 7.2, which concerns disposable acts. While reading the paper, I wondered whether an aligned reward also guarantees that an IA has an interest in being as precise as it can: if it knows the state lies in a small set F', is it always rewarded at least as much for reporting F' as for reporting a larger set containing it? The paper does not answer this question. A quick search suggests that it is studied elsewhere under the name of proper scoring rules for imprecise forecasts (e.g. Singh, Chau and Muandet, UAI 2025), which the paper does not refer to. It would be useful to state what "aligned" covers: preservation of the decision maker's preferences only, or also the IA's incentives.

Questions for authors

An experiment with non-singleton terminal acts (A ≠ {{ω}}), since handling an arbitrary A ⊆ F is presented as the main advantage over prior work and is never exercised. The medical example of Section 2 would be a natural candidate: what reward does Algorithm 1 produce for F = {C1, R1} under each criterion that is valid in that case (avg, min and reg according to Example 6.1), and what can be said for nd?

Corrections and suggestions (no effect on the score).

Example 2.1, which motivates the paper, is never completed: no reward is computed for F = {C1, R1}, although Section 2 presents it as the difficult case.

How would step 2 of Algorithm 1 (asking the decision maker for their criterion) be carried out in practice? Are there elicitation procedures that could be used or adapted, for example Lopes de Souza, Destercke and Imoussaten, "How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure" (arXiv:2609.33352)?

de Souza et al. (2025), on explainable evidential clustering, is cited among the works that define U directly from the cardinality of the set; its relevance to set-valued rewards should be made explicit.

References: typo "Didler Dubois"; Sutton and Barto (1998) has no publisher; Christiano et al. is dated February 2023 (the date of an arXiv revision) although the work appeared at NeurIPS 2017.

1 comment
  • We thank the reviewer for their feedback. Regarding the motivation and ML application concerns, we tried to address those through new experiments that we detail in the general response.
  • We thank the reviewer for raising the question of compatibility with the nd criterion. In fact, the problem is that the non-dominated criterion is a "too general" strategy, and since any aggregation imposes some extra structure on the reward, it is not possible to preserve such preferences. We added a proof in Appendix B that no symmetric, idempotent aggregator guarantees nd-alignment for every utility under which nd is valid. We believe this clarifies the limits of the construction.
  • The concern about identifying the criterion is indeed relevant, we acknowledge that asking the DM to identify it directly may be impractical. We appreciate the suggestion of using elicitation methods such as Lopes de Souza, Destercke and Imoussaten, "How Much Imprecision is Enough Imprecision in my Classifier? A Practical Elicitation Procedure" (arXiv:2609.33352). The problem is not exactly the same, but its general structure could be useful: after finding compatible aggregators, we could present examples where their induced choices differ, and then use the DM's choices to select the reward. This could then be integrated into a classification task, similarly to that work. Their procedure could also help elicit the initial utility uu when non-singleton terminal acts have utilities, instead of simulating these utilities as in our experiments. When uu is defined only for singleton terminal acts, classical utility elicitation methods are available. We added the citation to the manuscript.
  • "I wondered whether an aligned reward also guarantees that an IA has an interest in being as precise as it can: if it knows the state lies in a small set F', is it always rewarded at least as much for reporting F' as for reporting a larger set containing it?": Yes! By Theorem 7.2, no report (epistemic act) is disposable, so under information F′F', no other report is strictly preferred to F′F' under the chosen criterion. Since the criterion induces a weak order, F′F' is weakly preferred to every larger available report. Then the incentive to be as precise as possible does exist (a consequence can be seen in the application example that we now added, see Figure 2, Section 9).
  • We defined alignment as the property of a reward that preserves the preferences implied by the DM's utilities and criterion. If every terminal act deemed reasonable for such an epistemic act is preferable (or preferred) to some other act via the DM's utilities and criterion, then this epistemic act should itself be preferable (or preferred).
  • We thank the reviewer for pointing us to Singh, Chau and Muandet (UAI 2025). We had missed this reference and have now added it to the related work. They indeed share our concern for trustworthy reporting, proposing a method in the imprecise-probability framework. Their properness requirement is closely related to our non-disposability. The main difference seems to be that they rely on a randomized method that does not assure our alignment property.
  • We thank the reviewer for the typo and citation corrections, we have corrected these in the paper. de Souza et al. formulate a utility matrix over 2Ω×2Ω2^\Omega \times 2^\Omega, which is not exactly but relates to our setting. They use it, with a parameter λ\lambda controlling the cautiouness reward, in implementing a clustering algorithm. We added a description of this and the other related methods in Appendix D.
  • We hope these changes address all the points raised. We would welcome further questions or feedback, especially if any clarification is still needed.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General answer about the experiments

  • We thank the reviewers very much for highlighting the need for practical validation. We have expanded the experiments as follows:
    • We revisited the medical utility example to illustrate the impact of the rewards on the IA's behaviour. Table 5, Section 9 illustrates this, showing the "deceiving behaviour" that a naive approach generates. In Figure 2, Section 9, we tested these utilities on a proper ML problem, using the Heart Disease dataset as a practical medical example. Notably, our proposed aligned rewards substantially reduced the excessive abstention of naive approaches. In Appendix E.2 we show some further results.
    • We also extended the previous (and the new) experiments to non-singleton matrices (see Appendix E.4). Adding non-singleton terminal acts confirms that naive aggregation can still discard acts, while the matching aligned construction avoids this.
    • We also extended the random utility experiments to sampled matrices in the Heart Disease dataset, see Appendix E.3. And conducted a benchmark across 6 other datasets. The aligned rewards yield a much more trustworthy IA behaviour, without significant differences for utility maximization.
    • We also repeated the reporting comparisons on randomly sampled probabilities (Appendix E.6), showing the general behaviour of the machines that follow different rewards.

The experiments use fixed classifier probabilities to isolate the effect of reward design on reports and downstream decisions. All code, and some additional results, are still available at https://anonymous.4open.science/r/aligned-reward-under-uncertainty_exps-F0D3/.

The reviewer thanks the authors for their responses, for clarifying the points raised in the comments below, and for sharing the new experimental results.

The additional experiments cover several datasets, but the utility matrices remain largely synthetic, and the comparisons focus mainly on naive aggregation-based rewards. Could the authors explain why these baselines are sufficient to demonstrate the practical benefits of the proposed framework? More generally, what do the experiments demonstrate beyond addressing the known limitations of naive aggregation?

We thank the reviewer for the follow-up. We believe the experiments illustrate two phenomena.

  1. A naive strategy creates disposable acts
  • Our first experiments, now in Appendix E.4, show that the naive strategy (i.e., aggregating the utilities of the elements of AA to obtain its reward) often creates disposable acts. The compatibility results and Theorem 7.2 guarantee that our aligned construction avoids disposable acts. We aimed to test whether this is indeed a problem for the standard naive methods (or if it were instead restricted to a few artificial examples).
  • The experiments show that this is a recurring problem, even when more elaborate aggregators are introduced. To the best of our knowledge, all prior work that addresses this problem uses the naive strategy and focuses on the choice of aggregator. Our contribution shows that this is not enough to ensure alignment.
  • We emphasize that by naive rewards we do not refer to a specific aggregation method, but rather to a general approach that can use any aggregation operator. Our theoretical comparison with this approach applies regardless of the choice of aggregation operator.
  • The choice of using randomly sampled utilities intended to remove any bias that the examples we proposed before might have introduced. Conducting a utility elicitation experiment with a decision-maker would be difficult within the scope of the present article. This will be a subject of our future work.
  1. Disposable acts generate a deceiving behavior
  • We added the other experiments in response to the reviewers’ request for empirical evidence of how alignment affects the IA’s reports and the resulting decisions. We acknowledge that this evidence was missing, and focused on providing it in the final version of the paper.
  • Our tests on Example 2.1 illustrate this benefit. Table 5 shows what we mean by the deceiving behavior induced by the existence of disposable acts: the IA withholds information. A disposable act is not chosen even when it describes the IA’s assessment exactly. In trying to maximize the DM’s utility, the naive IA can gatekeep information that would be useful to the DM. For both real datasets and randomly sampled probabilities, this means that the IA’s output will be greatly distorted from the baseline. In the name of maximizing the DM’s utility, the machine will never output some states of the world. This is paternalistic behavior that harms the trustworthiness needed for human-machine interaction.
  • We varied the utility matrices, datasets, and probability inputs to check whether the results depended on a particular setup. Across all six benchmark datasets, aligned rewards drastically reduced this misalignment without harming the DM’s utility maximization. With our algorithm, the IA’s more faithful reporting comes at little or no cost to the DM’s realized utility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.