acceptodds
Under review as a conference paper at ICLR 2027

Decision-Relevant Predictions with Joint Scoring Rules

Abstract

When a decision-making principal wishes to elicit predictions from an agent using a proper scoring rule, the principal must specify the outcomes to be assigned probability. This can result in the principal either missing important considerations or being given too much information to process. We would instead like the agent making the predictions to provide the decision-relevant set of outcomes. This could be either the simplest set of outcomes leading the principal to the decision they would make under full information, or the simplest set of outcomes where no further information would change the principal's decision. We show that it is impossible to strictly incentivize a single agent to provide either of these sets, but that it can be done with two or more agents evaluated jointly. This is achieved by rewarding agents for refining the outcome set in a way that changes the principal's decision. We then show how to adapt the mechanisms to elicit a prediction alongside the decision-relevant outcome set, for cases where the set of possible outcomes is very large or continuous. Applications are demonstrated, first with a simple experiment where agents reveal a minimal mask of an MNIST image that allows a judge to classify it, and then by using language models to generate a small set of unit tests that allow a weaker judge to accurately evaluate code.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
predictions, proper scoring rules, ai alignment, decisions

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xPj7OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates how to incentivize agents to provide decision-relevant predictions using joint scoring rules. The authors establish theoretical limitations for single-agent settings and propose mechanisms involving multiple agents. The approach is evaluated through experiments on MNIST and a code evaluation task using language models.

Critical strengths

The paper addresses an interesting problem and provides theoretical contributions regarding the limitations of single-agent mechanisms and the potential of joint scoring rules. The proposed framework is potentially relevant to decision-making applications.

Critical weaknesses

The paper is difficult to follow, and the presentation is not always well organized. For instance, Section 3 is titled "Results" but still introduces methodological contributions. More importantly, the motivation for using agents rather than simpler models or approaches is not sufficiently clear. The experimental evaluation is also limited. MNIST is a relatively simple dataset, and experiments on a single dataset provide insufficient evidence of the approach's generalizability. Furthermore, the dataset used in Section 4.2 is not clearly described.

Questions for authors

Could the authors better justify the need for agent-based mechanisms compared to simpler alternatives? How do the proposed mechanisms compare with such approaches? Could the authors clarify which dataset is used in Section 4.2 and provide stronger experimental evidence supporting the generalizability of their findings?

3 comments

Thank you for taking the time to engage with our paper. Our aim is for the paper to be accessible to readers from a wide range of backgrounds, and your feedback is helpful for identifying where to provide further explanation.

The introduction has been re-written to include an example with a decision maker trying to maximize their expected utility, along with a visualization of the problem. It shows how truthful predictions can lead to suboptimal decisions, and why eliciting the right set of outcomes is challenging.

The paper's focus on agentic optimization allows us to establish formal guarantees about the reports that will be provided in equilibrium. This is akin to the proper scoring literature, which covers how honest and accurate predictions can be incentivized through the choice of reward. Our main contribution is defining reward structures that additionally incentivize predictions to be simple and decision-relevant, even when a principal lacks knowledge of what to elicit. In addition to the theoretical results, the precision of this approach allows it to outperform standard prompting methods on both simplicity and usefulness, as shown in our second experiment. If you have in mind other alternatives to agent-based mechanisms, we can provide further comparisons.

At your suggestion, we have renamed Section 3 from “Results” to “Mechanisms and Results”.

We agree that MNIST is a simple dataset, and we chose it for the first experiment as a proof-of-concept providing intuition, as well as an environment that would be easy for other researchers to build on. The second experiment uses the EvalPlus dataset, which contains examples of LLM generated attempts at solutions to 164 coding problems from the HumanEval dataset. We have moved additional details regarding that data from the appendix to the main body. While MNIST alone would not provide strong evidence of generalizability, the second experiment shows that the mechanism can be used to train and prompt LLMs on a task with practical applications.

We hope that these updates help clarify the motivation and contribution of our research, along with how the theory ties to experiments, and we ask that you please reconsider your rating of the paper. If you have specific concerns regarding the soundness of our results, we would welcome the opportunity to address them.

I thank the reviewer for their rebuttal but my score remain unchanged.

Thank you for the fast response. Are there concerns you raised in the initial review that you feel are still unaddressed? Understanding which points, if any, you continue to find unclear or insufficient would help us determine whether further clarification is needed.

Reviewer_CZKnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines a setting in which the principal wants to make a decision without processing the full detailed prediction. So the question this paper seeks to answer is: how does this principal delegate the selection of information to an expert so that the principal can process as little information as possible while still making the same decision they would make with complete information? To answer this question, the authors design a mechanism in which one expert provides a simplified report, and another expert can add details that were left out. The authors characterize what reports the experts would choose when each acts to maximize their own reward. Under the model’s assumptions, the authors prove the existence of a subgame-perfect equilibrium. Numerical studies show that the proposed approach can reduce the amount of information provided while retaining high accuracy in some settings.

Critical strengths

This paper studies an important and interesting question: how a principal can ask experts to select a small amount of information while preserving the decision they would make with complete information. By allowing experts to challenge each other’s reports regarding omitted details, the paper addresses the difficulty of checking whether a simplified report omits something important.

Critical weaknesses
  1. It is hard for me to establish a clear chain of problem-consequence-solution-evidence. If I understand correctly, based on the introduction, this paper seeks to answer how this principal delegates information selection to an expert so that the principal can process as little information as possible while still making the same decision they would with complete information. To answer this question, the authors design a mechanism. However, I failed to see what consequences would follow without adopting the proposed mechanism, or why simpler approaches would be insufficient. And I seek the experimental demonstration to help me understand the application and the necessity of the proposed mechanism. But what I can learn from the experimental results is only that models can sometimes provide fewer pixels or tests, and I failed to see the evaluation showing that the proposed mechanisms prevent consequential omissions or preserve the full-information decision. Moreover, several code-training results show declines in accuracy, which again do not align with the motivation and the mechanism.

  2. My other concern is the way of simplifying the information provided by the expert. I think this paper tried to reduce the amount of information the decision maker has to process. Under certain assumptions provided in this paper and its detailed proofs, I believe the proposed mechanism can, to some extent, encourage experts to provide the simplest report that meets the specified decision requirements. However, the paper does not discuss the effort experts spend finding that report, the cost of multiple rounds of interaction, or the burden of asking the decision maker to evaluate reports repeatedly. As a shorter final report does not necessarily mean less work overall, I am not convinced that the proposed mechanism is more efficient in practice or that it is needed. Besides, when I read the mechanism and try to understand the procedure, I would naturally ask what the limitation of the mechanism is that asks two experts to analyze it independently, critique each other’s findings, and then produce a concise report, especially in this paper, the cost is not the constraint that is taken into consideration.

  3. As I cannot find a clear mapping between the model and the experimental setup, it is hard for me to evaluate the value of the experimental studies. Besides, the authors mention in the Discussion section that the proposed mechanism can be used to understand disagreements and identify what AI systems value. The disconnect between the introduction and discussion makes me confusing about this paper's main contribution.

Questions for authors
  1. I would appreciate it if the authors could provide a concrete example to show how the proposed mechanism changes the outcome. This explanation can help me to understand the necessity of designing such a mechanism, considering the motivation mentioned in the introduction. Furthermore, what's the challenging aspect of designing such a mechanism?

  2. How does the proposed mechanism differ from other mechanisms, such as asking two experts to analyze the inforamtion independently, critique each other's findings, and then produce a concise report? As sequential critique alone does not establish the novelty of the approach, could the authors clarify where the main technical contribution is?

  3. Can you provide a more detailed discussion mapping the experimental test to the theoretical objective? For each task, what is the principal's decision rule? What corresponds to the full-information report? How to understand the information exchange between two experts in the experimental setting? Furthermore, can the experimental results show that the proposed mechanism reduces the principal's burden or total computational cost?

  4. Does the reward mechanism add any assumption on the experts, especially the second expert? Based on my understanding, if the second expert adds information that changes the principal’s decision, the second expert receives a bonus and the first expert is penalized. Given that the second expert has access to complete information and observes the first expert’s report, there exists a possibility that the second expert strategically selects additional information to induce a change in the decision and earn a bonus that could exceed the simplicity reward. Can the designed mechanism rule out this behavior? If so, how?

3 comments

Thank you for your detailed review, which has helped us identify many improvements.

Problem-consequence-solution-evidence: P - Powerful agentic systems have knowledge that could be elicited but that we do not know to ask about C - Humans make suboptimal decisions based on incomplete information S - The second opinion and rebuttal mechanisms elicit all decision-relevant information, without extraneous details E - Proofs and experiments showing that AIs can be trained/prompted to provide simple and informative sets of pixels/unit tests

Questions: 1: We have re-written the introduction to include an example where a utility-maximizing decision maker chooses whether to deploy new software. Grouping minor and major failures leads the principal to go ahead with deployment, but distinguishing severity reverses that decision. The example then shows the challenge of having an agent simplify, tied to our impossibility result, which is that evaluating a partition of outcomes is insufficient for determining if it is missing relevant information.

4: The possibility of the second expert strategically selecting additional information to induce a change of action is a critical part of the incentive structure in the second opinion mechanism. If there exists a truthful refinement that would change the decision, the first report must be omitting decision-relevant information. Anticipating that the second expert would be rewarded for such a refinement is what gives the first expert an incentive to preemptively provide all relevant information. Different experts can be heads on the same neural network or copies of the same model, and so we assume access to the same information.

The insight of our mechanisms is that we can design a multi-agent equilibrium incentivizing behavior that would not be possible with one agent. In that equilibrium, the first expert reports everything relevant, followed by the second expert repeating that report. This prevents a need for generating and evaluating a new partition, and after training only the first expert needs to be run.

Since the agents in training are supposed to change actions away from the correct one when responding to an oversimplified partition, small drops in accuracy over some training curves are expected and not a cause for concern. The results table in the appendix removes these cases from the evaluation, and only one model shows a statistically significant drop in accuracy (Nemotron 3 30B).

2: The fundamental issue is that a decision maker alone cannot evaluate whether a concise report contains all decision-relevant information, and so cannot incentivize it. The novelty of our approach lies not in sequential critique, but rather in developing formal scoring rules to produce equilibria satisfying precise simplicity and decision-relevance criteria. Asking experts to produce a concise report without oversimplifying is otherwise underdetermined, which motivated our work on the problem.

A further challenge with many alternatives is that experts may lie or deliberately make arguments hard to understand, especially when rewarded for convincing decision makers. The mechanisms we introduce circumvent this by restricting “arguments” to partitions of predictions elicited with proper scoring rules, which other approaches could also incorporate.

3: The section on experiments now clarifies the elements of the theoretical model.

We have added a new Appendix C discussing effort costs. Multiple rounds of interaction and repeated evaluations only apply in training until equilibrium is reached. In some applications, such as our second experiment, the theoretical model allows for precision in prompting, which avoids such costs entirely.

You are correct that a smaller partition does not necessarily mean less work for the principal if the coarsenings are difficult to process. Although we did not see such results empirically, they could be patched by disallowing partitions that group unrelated outcomes together. For example, in the coding experiment, we could reject unit tests that use an OR statement to combine evaluations. In that experiment, reading fewer unit tests corresponds to fewer input tokens and so lower computational cost for the judge.

Other Concerns: We have added tables to the appendix showing how well the partitions in the MNIST experiment avoid omissions and preserve the full-information decision. After a minor fix increasing the penalty for omissions, 55.5% of partitions for individual MNIST images are robust to white box adversarial examples, despite not being trained against them. This rate increases to 90.6% when training using a lower distance threshold for action changes. A similar analysis could be run on the coding experiment if it would meaningfully inform your assessment, though we would expect analogous results.

If these revisions address your concerns, we ask that you please consider raising the assigned rating. We welcome any remaining questions.

Thank you to the author for the reply and for adding the illustrative Figure 1. However, it is still a bit hard for me to connect the practical motivation with the experimental results. If my understanding is correct, the simplification report is mathematically modeled as partitions. However, in the experiments, simplicity is reflected in fewer revealed pixels or fewer unit tests. It remains unclear to me how closely they capture the reduction in the principal's processing burden that motivates the paper.

Following this confusion, I raise the question of costs incurred. Although Appendix C discusses the expert effort and costs, it does not clarify my concern about whether the proposed method is better overall than letting the principal process the full information directly. Hence, the motivation for practical efficiency benefits is contingent on your particular setting and model.

Furthermore, it remains unclear how the theoretical assumptions concerning the second expert (e.g., the ability to find profitable refinements) are realized or approximated in the experiments. For example, what happens when the second expert fails to identify an important omission?

Thank you for further clarifying your concerns. We believe that we can address each of the outstanding points.

You are correct that agent outputs correspond to partitions. In the MNIST experiment, a pixel mask partitions images into groups that agree on the revealed pixels. In the coding experiment, test suites partition programs by their outputs on the given tests.

The MNIST example builds on the paper AI Safety via Debate (Irving et al., 2018), which established using pixels revealed as an analogy for more complex arguments. We agree that outside of this analogy, a human or AI looking at a smaller pixel mask may not have a lower cognitive burden. However, we believe the coding example is more directly useful and illustrative. Fewer unit tests generally require fewer tokens and thus less compute for our non-reasoning judge. With a human judge, each test clearly takes cognitive effort to read and understand, so lowering the number of tests will generally reduce total burden, even if there is some variance in test difficulty

The coding example also shows the difficulty with having the principal directly evaluate the full information partition of program behavior. Processing every possible unit test (perhaps filtered for length or other criteria) would take an enormous amount of time and cognitive effort. The introduction example can be understood similarly. While we split “failure” only into “minor failure” and “major failure”, full information could easily involve hundreds of distinct failure modes. Practical benefits from simplifying will depend on the setting, but applications where full information is very costly to process are widespread.

While reducing cognitive burdens is a motivation for this research direction, the primary motivation is to first elicit all decision-relevant information that the agent knows, and only then optimize simplicity. Full information trivially achieves the first part of the objective, but in many applications is impractical to process. If there is a good model of the principal’s cognitive costs, it can be used as the simplicity measure, but otherwise proxies such as the number of outcomes can still address concrete sources of processing burden. In rich real-world settings, almost all possible distinctions between outcomes are irrelevant to almost all decisions. Any simplicity measure then supports leaving them out, so different measures will closely track each other even when they disagree on which partition is the very best.

Regarding the second expert, we are primarily concerned with eliciting decision-relevant information known by the agent that the principal would not think to ask for. If the agent is not aware of an important omission, then no mechanism can elicit it, but the principal does not fail to learn what the agent knows. Our training process is based on the memoryless second opinion mechanism with one model playing both experts, so the second expert has the first expert's information by construction. Multiple potential partitions are generated at each training step, and refinements that expose omissions receive higher rewards, resulting in the agent being taught both about the task and about what constitutes an important omission.

We have edited the paper to incorporate these clarifications. If they address your remaining concerns, please consider raising your rating.

Reviewer_QiefOpenReview ↗
Rating
4
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies the information elicitation problem, where the agent reports a partition of states and predictions over the partitions. The motivation is that, when the state space is high-dimensional, it may be untractable for the agents to report a probabilistic forecast. The main results are:

  • When the distribution over states is known to the principal, the optimal partition cannot be elicited from a single agent. However, by using a second agent to audit the first agent's report and rewarding the marginal gain of the second agent, both are incentivized to report the optimal partition truthfully.

  • When the distribution over states is unknown and the agents select a partition with a prediction over the partitions, the same idea of auditing from a second agent applies to a truthful mechanism too.

  • The paper uses the mechanism for prompting and training on the MNIST dataset and the unit test selection application. These provide evidence and concrete motivations of the paper.

Critical strengths

I really like the idea of the proposed mechanism. The proposed mechanism is simple and easy to understand. It connects proper scoring, debate, and second opinions in a clean way. The experiment validation also provides a concrete motivation of the problem.

Critical weaknesses

I was a reviewer of a previous version of the paper. My main concerns about the applications were resolved in the current version. I only have minor concerns about the theoretical approach.

  • The interpretation of the decision rule remains unclear. Although the paper provides experimental examples, it does not explain how the decision rule relates to a rational or risk-averse principal. The relationship between the simplicity-based reward and the principal’s gain from the decision is also unclear. A concrete utility-based example would help explain the model and clarify whether the single-agent impossibility still holds under natural restrictions on the decision rule.

  • The necessity of a second agent is not established when predictions must also be elicited. Theorem 3.1 considers the setting where predictions are given, whereas Section 3.3 allows agents to report both a partition and predictions. The positive two-agent result does not establish that a single agent cannot achieve the same objective under the same assumptions and honesty requirements. A corresponding impossibility result, or a precise argument explaining how the earlier impossibility extends to this setting, would help justify the need for a second agent.

Questions for authors

It would be great if the authors could expand on my comments on weaknesses.

1 comment

Thank you for your in-depth engagement with our research, and for the strong recommendation.

We chose the form of the decision rule for generality, but agree that your suggestion of a concrete examples would help with exposition. The introduction of the paper has been updated to add a simple case with a utility maximizing decision-maker, along with a visualization of the problem. A risk-averse decision maker, maximizing an objective like sqrt(utility), would choose actions similarly.

At the end of that section, the utility maximizer example is used to provide intuition for the impossibility result. This comes from the decision maker having beliefs about the distribution of outcomes within a coarse set, which will sometimes be accurate and other times not. Coarsening should only be rewarded when inaccurate beliefs do not affect the decision, but this cannot be verified when they concern untaken actions.

The impossibility result has been extended to the case where predictions must also be elicited, in a new Theorem 3.6. This uses the fact that the optimal partition will sometimes be the maximal coarsening (only one outcome, with probability 1), and the prediction does not matter in that case.

We have added an Appendix C, which expands on models of cognitive costs and considerations around that. Decision-relevant predictions reduce cognitive costs from the full information case, but do not account for when the principal would prefer to take a suboptimal action if it saves enough cognitive effort. Under some assumptions about the principal's utility function and decision making process, the cognitive cost can be incorporated into the distance metric so that refinements are only provided if the expected utility gain is greater than the cost.

If you feel that these additions to the paper address your remaining concerns, we ask that you please consider increasing the confidence of your rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.