Decision-Relevant Predictions with Joint Scoring Rules

When a decision-making principal wishes to elicit predictions from an agent using a proper scoring rule, the principal must specify the outcomes to be assigned probability. This can result in the principal either missing important considerations or being given too much information to process. We would instead like the agent making the predictions to provide the decision-relevant set of outcomes. This could be either the simplest set of outcomes leading the principal to the decision they would make under full information, or the simplest set of outcomes where no further information would change the principal's decision. We show that it is impossible to strictly incentivize a single agent to provide either of these sets, but that it can be done with two or more agents evaluated jointly. This is achieved by rewarding agents for refining the outcome set in a way that changes the principal's decision. We then show how to adapt the mechanisms to elicit a prediction alongside the decision-relevant outcome set, for cases where the set of possible outcomes is very large or continuous. Applications are demonstrated, first with a simple experiment where agents reveal a minimal mask of an MNIST image that allows a judge to classify it, and then by using language models to generate a small set of unit tests that allow a weaker judge to accurately evaluate code.