acceptodds
Under review as a conference paper at ICLR 2027

Evidence Markets

11 views

Abstract

Current prediction markets face two limitations that restrict their broader applicability: (1) they reveal what the crowd believes but not the evidence or reasoning behind those beliefs, and (2) they require an exogenous event with an external ground truth that resolves at a known future date. We address these twin challenges by introducing evidence markets, a generalization of prediction markets that (1) incentivizes the submission of evidence alongside beliefs and (2) can resolve endogenously from that evidence when exogenous resolution is unavailable. At its core is a logarithmic market scoring rule whose liquidity parameter decreases as accumulated evidence quality increases. We prove that platform loss here is bounded, that evidence is rewarded in proportion to current market uncertainty, and that the mechanism admits an equivalent automated market maker. Importantly, truthful belief and full evidence reporting is a dominant strategy equilibrium under exogenous resolution and an -subgame perfect Nash Equilibrium under endogenous resolution, where we also bound how withholding evidence shifts a trader's belief about resolution. Throughout the work, LLM evaluations–determining which model is best for a given task–is used as a running example for such a market.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prediction markets, llm evaluation, mechanism design, algorithmic game theory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_rBfSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes evidence markets, in which traders report both beliefs and supporting evidence. It extends the logarithmic market scoring rule by making its liquidity parameter decrease with the quality of accumulated evidence. The resulting payoff rewards informative evidence, with an entropy-dependent component. For exogenous resolution, the authors claim dominant-strategy truthful belief reporting and full evidence disclosure; for endogenous resolution, the outcome is sampled from a softmax of scores computed on submitted evidence, and the authors claim an approximate subgame-perfect equilibrium. The paper also gives a cost-function AMM representation, a bounded-loss argument, and practical sketches for verification and asynchronous execution. LLM evaluation is the principal motivating application. The evaluation is entirely theoretical, with no experiments or empirical validation.

Critical strengths
  1. A relevant and appealing problem formulation. Incentivizing participants to disclose the evidence behind their forecasts could make prediction markets more interpretable and create reusable artifacts, including evaluation questions for LLMs. Treating evidence as an explicit economic input, alongside beliefs, is a worthwhile research direction.

  2. A parsimonious mechanism with an interpretable payoff. Evidence-dependent liquidity yields a simple connection between the reward for evidence and market uncertainty. The bounded-loss calculation and cost-function AMM representation make the proposal concrete. In the exogenous setting, the basic full-disclosure incentive is particularly transparent.

Critical weaknesses
  1. A substantial gap between the model and the proposed LLM-evaluation application. The model endows each trader with a fixed evidence set and analyzes selective disclosure; it does not incentivize the creation or acquisition of useful evidence. It also assumes a monotone, effectively verifiable quality measure. More importantly, endogenous resolution evaluates models on the evidence voluntarily submitted to the market. Full disclosure of these privately selected sets does not imply that the collected questions are representative of any target task distribution: an entirely valid but systematically selected set may reverse the ranking of models on that distribution. This is not merely an implementation detail, because the paper motivates endogenous resolution as a way to answer which model is better at a task. The authors should either identify assumptions connecting submitted evidence to the target evaluation objective, or narrow the claim to ranking models on the collected evidence.

  2. The incremental technical contribution of evidence-dependent liquidity is insufficiently established. Many guarantees follow directly from familiar ingredients: properness of the logarithmic score, monotonicity of the evidence-quality function, telescoping payments, and softmax sensitivity. As a comparison point, a standard constant-liquidity LMSR plus a bounded, monotone cumulative evidence bonus already yields truthful belief reporting, full disclosure of pre-existing evidence, and bounded platform loss under exogenous resolution. The paper does not establish which economically meaningful property specifically requires evidence-dependent liquidity, rather than a separable evidence reward. The approximate endogenous incentive guarantee also relies on increasing softmax temperature, thereby making resolution less informative. A formal separation from the simple baseline or a quantitative incentive-versus-informativeness result would make the contribution significantly more compelling.

Questions for authors
  1. For LLM evaluation, what is the target task distribution? What ensures that the strategically selected evidence is representative of it? If no such assumption is made, is the intended output only a ranking on the collected evidence?

  2. Which guarantee specifically requires evidence-dependent liquidity? A conventional LMSR plus a bounded, monotone evidence bounty also supports truthful belief reporting and full disclosure under exogenous resolution. Can the authors show a formal separation or an additional property?

  3. Theorem 5.1 uses a lower bound on the previous report seemingly derived from truthful reporting. Since SPNE covers off-path histories, how is this bound ensured for arbitrary admissible past reports? Is a truncated report space intended? I regard this as potentially fixable.

2 comments

We thank the reviewer for the careful review and for recognizing the strengths of our work: the relevance of eliciting evidence alongside beliefs, and the parsimony and interpretability of the mechanism. The reviewer notes that evidence-dependent liquidity “yields a simple connection between the reward for evidence and market uncertainty.” That parsimony, and that connection, are precisely part of our answer to the question of incremental contribution below.

Representativeness of the collected evidence: This is a mechanism-design paper. Its contribution is a mechanism that lets traders convey why they trade, by incentivizing full disclosure of evidence alongside beliefs, and that can resolve from that evidence when no external resolution exists. It is agnostic to the quality function: the discriminative score in Appendix C is one instantiation, and our guarantees hold for any non-negative, monotone rr (Definition 3.2)

Every assessment of whether one model is better than another relies on a selected dataset, whose relationship to the capability of interest depends on how it was constructed. This holds for static or dynamic benchmarks and is not specific to our setting. What our mechanism adds is that traders are incentivized to disclose all the evidence they hold, including evidence that cuts against their own position (Theorem 5.1).

Admission criteria connecting evidence to a target task are the market proposer’s design choice, encoded in rr and its verification regime (Definition 3.2, Appendix C), e.g., relevance to a task specification, coverage across categories, or exclusion of near-duplicates. The mechanism’s incentive guarantees are domain agnostic. We make this division of responsibility explicit in the revision.

Contribution of evidence-dependent liquidity: We do not claim it is the only route to the exogenous guarantees; a constant-liquidity LMSR with a separate bonus can also deliver them. The question is what the design achieves beyond them: Parsimony within the same budget. As the reviewer notes, the designer specifies a single schedule β(⋅)\beta (\cdot). Worst-case loss remains β(R0)log⁡n\beta(R_0) \log n,identical to a plain LMSR (Proposition 5.1): evidence rewards are funded within the liquidity subsidy and spent only as evidence arrives. At a fixed budget, a separable bonus adds a second parameter, lowering liquidity on every history, and thus reducing the sharpness of information aggregation .

Importantly, evidence in our setting is priced by remaining uncertainty. The expected evidence payoff is xH(qt)xH(q_t) (Lemma 5.1, where HH is the Shannon entropy): evidence earns most when the market is uncertain and nothing once it is settled. This follows directly from the single schedule, not from a separately designed weighting.

Lastly, our design dampens free-riding on public evidence. Suppose a trader submits evidence but does not trade, leaving the market price unchanged; this is the risk-free, evidence-only trade of Section 5.1. A later trader can read that evidence and move the price accordingly, collecting the belief-update reward for information they did not supply. Under constant liquidity, that later trader collects the full reward. Under our design, the submitted evidence has already lowered liquidity, so the later trader’s reward shrinks and part of the subsidy goes to the evidence supplier instead.

On the softmax temperature: A larger τ\tau affects only the sharpness of the resolution lottery, not the information collected. Since full disclosure is incentivized, any function of the cumulative evidence (e.g., the argmax of scores) can be reported alongside the market independently of τ\tau (Section 4). The trade-off is between lottery sharpness and the incentive bound, not between incentives and the informativeness of the collected evidence.

Lower bound in Theorem 5.1: We thank the reviewer and adopt the suggested truncated report space. Restricting reports under endogenous resolution to beliefs bounded below by pmin⁡p_{\min} keeps the truthful report admissible, since every resolution belief already satisfies this bound (Lemma 4.1), so Proposition 5.1 and the rest of the proof are unchanged. The revisions are updated in the paper.

Thank you for the detailed response. I appreciate the clarifications and the additional pilot study. The proposed restriction on the report space also appears to address my concern regarding Theorem 5.1.

However, my two main concerns remain.

First, while representativeness can reasonably be delegated to the market designer, full disclosure of existing evidence does not guarantee reliable model rankings under the target task distribution. The pilot study does not establish this connection.

Second, the incremental technical contribution of evidence-dependent liquidity remains unclear. The proposed advantages are interesting, but no formal separation from a constant-liquidity LMSR with a bounded, monotone evidence bonus is established. In fact, the same softmax sensitivity argument appears sufficient to obtain an approximate endogenous incentive guarantee for this simpler baseline under comparable assumptions.

Overall, I appreciate the authors' clarifications, but these concerns remain substantial. I therefore maintain my original rating of 2.

Reviewer_ENunOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Evidence Markets, a generalization of prediction markets where participants can submit both their beliefs and supporting evidence. The main motivation is that traditional prediction markets do not reveal the evidence behind participants' predictions, and they usually require an externally verifiable outcome for resolution.

The authors propose a modified logarithmic market scoring rule (LMSR), where the liquidity parameter decreases as the accumulated evidence quality increases. They show that the mechanism has bounded platform loss and provides incentives for truthful belief reporting and complete evidence submission. For exogenous resolution, the authors establish a dominant strategy equilibrium, while for endogenous resolution they establish an approximate subgame perfect Nash equilibrium under additional assumptions. The paper also presents an equivalent automated market maker formulation. The paper is theoretical and uses LLM evaluation as its main motivating application.

I should say that mechanism design and prediction markets are outside my main area of expertise. I cannot fully assess the novelty of the equilibrium guarantees relative to existing work or verify all of the proofs. My assessment therefore focuses mainly on the proposed ML application and the practical assumptions behind it.

Critical strengths

The motivation is the main reason I see value in this paper. Rewarding participants for submitting evidence, rather than only predictions, addresses a real gap. LLM evaluation is a good example of a setting where an objective ground truth is hard to establish. If a mechanism like this works in practice, the collected evidence would be useful on its own, beyond the market price.

I also think it matters that the authors go beyond a conceptual proposal and give a concrete mechanism with formal guarantees. The construction builds directly on the LMSR, so its connection to existing work is easy to follow, and the paper covers both exogenous and endogenous resolution. The authors are also open about the difficult parts, including evidence verification and strategic withholding. They show with a herding example (Proposition 4.1) why the endogenous case cannot get a dominant-strategy guarantee. All of this makes the proposal something that can actually be evaluated, which is why I do not recommend a clear rejection.

Critical weaknesses

I think the paper doesn't explain whether the evidence collected by the market can be trusted, in terms of both quality and representativeness. The theoretical results assume a quality function that accurately reflects how informative the submitted evidence is. Deciding whether a piece of evidence is useful or correct is itself a hard problem. Appendix C proposes an LLM-as-a-Judge with staked disputes, but this component is not analyzed. It is unclear to me how reliable it would be when participants have money at stake and an incentive to manipulate it.

I also have concerns about the proposed application to LLM evaluation. In Appendix C, the authors suggest rewarding evidence based on how well it distinguishes different models. However, a highly discriminative evaluation question is not necessarily representative of the task distribution. For example, participants may deliberately select questions that favor one particular model. These questions could still be valid and receive high evidence quality scores, while producing a biased evaluation of the models. I would like to understand how the proposed mechanism addresses this issue.

Questions for authors
  1. How robust is the proposed evidence verification mechanism when participants deliberately submit misleading or adversarial evidence? 2. How would errors in the evidence quality function affect the incentive guarantees?
  2. In the LLM evaluation application, how can the mechanism prevent participants from selectively submitting valid but biased evaluation questions? I
1 comment

We thank the reviewer for their detailed review and insightful comments. We are glad they found the work well motivated and clearly presented, and that they valued its concrete mechanism and formal guarantees.

The concerns about evidence quality, verification and representativeness are quite important; however, they relate to how evidence is valued and verified, not the mechanism itself. None of our results assume a particular quality function (only assumed to be non-negative and monotone) or verifier. The discriminative score (Definition C.1) and staked disputes (Appendix C.1) are examples of instantiations, not requirements, and this flexibility is a strength of the design. Separating the mechanism from the valuation is standard in mechanism design: auction results, for instance, typically hold for a broad class of utility functions. This also answers Q2: errors in the quality function that preserve monotonicity change how much traders are paid but not what is optimal for them, so IC holds, and the platform’s worst-case loss of β0log⁡n\beta_0 \log n (Proposition 5.1) is unchanged. To elaborate in more detail:

Exogenous resolution: Here evidence affects only payments, never resolution, so a verification error can at worst shift payments. An LLM judge is a reasonable verifier for this role, and it improves as models improve. Our execution algorithm also does not depend on fast verification: orders execute at a pessimistic price, and traders are refunded once their evidence is verified (Theorem C.1, Appendix C.2). Verification can therefore take as long as needed, including after the event’s resolution, without slowing trading or changing anyone’s net payment. This helps against fabrication: submissions can be audited for falsity after the fact, when facts are often easier to check, before any submission stake (Appendix C.1) is returned. We do not claim to solve fabrication detection, a hard open problem in its own right; the mechanism is agnostic to the verifier and can adopt the state of the art as it improves. What it does bound is what a fabricated item can collect: it cannot change the outcome, and every evidence reward comes out of the same β0log⁡n\beta_0 \log n subsidy.

Endogenous resolution: Here the core issues are relevance and correctness. Staked dispute mechanisms like ours are well studied in distributed systems, including their failure modes and remedies. Getting a spurious, irrelevant or falsely answered question accepted would require fooling a frontier model at a verification task (generally easier than generation) and surviving the dispute process. This itself can be a useful datapoint if it occurs, but it seems harder than coming up with good questions. And a trader cannot fabricate the result: the outcomes XieX_{ie} are obtained by running each model on the submitted question (Appendix C.1), not reported by the trader, so under the discriminative score a verified question is paid only if the models really do perform differently on it.

Biased questions: We view this partly as a question of what an evaluation set is for. Its goal need not be a representative sample of the task but an adversarial one: if models score 95% on high school math, the informative evaluation concentrates on the 5% they get wrong, and questions one model fails and another passes are precisely what distinguishes them. Where the concern is a participant steering the outcome toward a particular model, Theorem 4.1 bounds, under history-independent play, how far any single trader’s submission can move the resolution, regardless of the questions it contains.

Pilot Study: We also have preliminary data on whether real participants supply good evidence and whether its quality can be assessed (not a test of the full mechanism; no trading, no evidence-dependent liquidity, points rather than money). LLM agents directed by 19 participants (not scripted to any strategy) submitted 815 pieces of evidence on 57 exogenously resolved binary questions about real-world events, each scored by an LLM judge using the quality function of Eq. 18 (Appendix C.1). First, the judge filters: it rejected 121 submissions (73 duplicates, 47 uninformative, 1 unverifiable) and gave zero quality to 228 more whose effect was within its noise. Second, participants supplied evidence worth paying for: 466 received positive quality. The pilot the two ingredients the theory takes as given, traders hold evidence and a quality function can assess it, are realistic. This is included in the revised version.

Evidence acquisition: While our mechanism is approx IC for traders to submit all their evidence truthfully, it does make a subtle assumption: traders already hold the evidence. In many cases, they may have to work to generate evidence, which may be costly. However, since the evidence bonus can be set by the market designer through the liquidity param β\beta, they can raise it for such costly acquisition settings.

Reviewer_JGdzOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Prediction markets are key to evaluating LLMs' forecasting ability. This paper addresses shortcomings of prediction markets: first, that prediction markets currently do not illustrate why people hold beliefs, and second, that events will resolve with time. The authors propose evidence markets as a generalization of prediction markets that addresses these shortcomings. The paper contributes the theoretical grounding for prediction markets but is limited on its immediate applicability to the LLM evaluation literature.

Critical strengths

The paper proposes an interesting extension of prediction markets that allows participants to submit evidence alongside their predictions. A key strength is that the proposed mechanism comes with clear theoretical guarantees: it incentivizes participants to report their true beliefs and submit their available evidence, while keeping the platform’s potential loss bounded. The ability to resolve a market using the collected evidence is compelling for settings in which there is no readily available ground truth.

Critical weaknesses

This paper reads more like a theoretical economics/mechanism-design paper than a machine learning paper. The authors motivate evidence markets using LLM evaluation as a running example, and I agree that this is a potentially compelling application. However, the connection currently feels largely illustrative: the main technical contributions concern the economic theory of a generalized prediction market, while the paper provides no empirical evaluation demonstrating that the proposed mechanism actually improves LLM evaluation in practice.

For an ICLR audience, I think the paper would benefit substantially from making the ML contribution more concrete. In particular, I would like to see empirical experiments with actual LLMs: for example, instantiating an evidence market in which submitted evidence consists of evaluation questions, and testing whether the resulting mechanism produces more informative or discriminative evaluation sets, better model rankings, or greater robustness than existing evaluation approaches. Such experiments would both validate the assumptions underlying the theoretical model and demonstrate that evidence markets provide practical benefits for LLM evaluation beyond serving as a motivating example.

Questions for authors
  1. How would you propose creating a benchmark using evidence markets?
  2. Do you foresee other applications of evidence markets in ML/LLMs besides evaluation benchmarks?
2 comments
AuthorsOpenReview ↗

Official Comment by Authors (I)

We thank the reviewer for their thoughts and comments. We are pleased that they found the work to be a novel extension of prediction markets with clear theoretical guarantees. We respond to their specific comments below:

Relevance to ICLR: We just want to emphasize that ICLR has a rich history of publishing works in algorithmic game theory and mechanism design which have applications to machine learning. Recent examples include Lin & Chen (ICLR 2025, spotlight), which characterizes the utility a principal can achieve against a learning agent in principal agent problems like contract design and Bayesian persuasion; Hossain et al. (ICLR 2025), which characterizes the equilibria of strategic classification with multiple interacting agents; Bacchiocchi et al. (ICLR 2024), which gives sample-complexity guarantees for learning optimal contracts; and Balcan et al. (ICLR 2026), which proves regret bounds for learning in Stackelberg games with side information. In each case, the main contribution is a set of formal guarantees about how strategic agents interact within a decision-theoretic framework. Our contribution is of the same kind. Importantly, our mechanism is aimed at a problem central to the ICLR community: obtaining fresh, discriminative evaluation data for LLMs in settings with no external ground truth. We therefore believe the paper is within the scope of ICLR.

Experimenting a real evaluation market: While a live deployment would be valuable, it is typically beyond the scope of a mechanism design paper. To test whether an evidence market produces better evaluation sets, one needs a working platform with real stakes and enough traders. Without that critical mass, an experiment cannot tell whether the market is working as intended. Simulated traders cannot stand in for this: agents scripted to follow the equilibrium strategy would only reproduce what our theorems already guarantee, while the real empirical question, how good the crowd-sourced evidence is, depends on real participants. All market design work shares this hurdle. For example, auction mechanisms for LLM advertising (Hajiaghayi et al., NeurIPS 2024) and interventions for data markets (Hossain & Chen, ICML 2024) were analyzed without being deployed to real advertisers or data buyers; but their contributions stand regardless. Likewise, our core contribution is the mechanism and its guarantees, which any future deployment would build on.

That said, we can share preliminary data on the empirical question above about whether real participants supply good evidence, and whether its quality can be assessed appropriately. Since submission, we ran a small pilot of the evidence-gathering and verification components to see how they might behave in practice. It was not a test of the full mechanism – there was no trading, no evidence-dependent liquidity, and participants competed for points rather than money. In this pilot, LLM agents directed by 19 participants (not scripted to any strategy) submitted 815 pieces of evidence on 57 binary questions about real-world events with exogenous resolution. Each piece was scored by an LLM judge following the quality function of Appendix C.1 (Eq. 18), i.e., by how much it shifts the judge's belief given the evidence already accepted. Two observations are relevant here. First, the judge filters: it rejected 121 submissions (73 as duplicates of evidence already in the market, 47 as uninformative, and 1 as unverifiable) and assigned zero quality to a further 228 whose effect was within its noise. Second, participants did supply evidence worth paying for: 466 submissions received positive quality. The pilot is small and does not test our equilibrium results, but it suggests that the two ingredients the theory takes as given, traders hold evidence and a quality function can assess it, are realistic. We have added the details to the appendix.

AuthorsOpenReview ↗

Official Comment by Authors (I)

Creating a benchmark with evidence markets: An elegant part of our model is that the benchmark is a direct consequence of the market itself! A market creator poses the event "which of models M1,…,MnM_1, \dots, M_n is best at task X?" and sets K (i.e., the number of evidence items needed to resolve). Traders submit evidence in the form of [question, answer] pairs. Each submission is verified for relevance to the task, correctness of the answer, and novelty relative to what is already in the market (Appendix C.1). For each accepted question, every model is run on it to obtain the outcomes XieX_{ie} (whether model i answered question e correctly), and the trader is rewarded according to how well the question discriminates between the models (Definition C.1). Once K questions are accepted, the market resolves from the scores θi\theta_i, the fraction of accepted questions each model answered correctly (Definition 3.4). The accepted questions are the benchmark, and the scores θi\theta_i give the ranking. Two features distinguish it from a static benchmark: (i) the questions are written after the models are released, so they are unlikely to have leaked into training data; (ii) traders are paid most for questions that separate the models while the market is still uncertain about which is better (Lemma 5.1), which works against saturation.

Applications beyond LLM evaluation: The mechanism applies whenever a question has to be answered from a body of evidence, and we see several applications within ML. (i) Red-teaming and safety evaluation: the same market can ask whether one model is more prone than another to an undesirable behavior (e.g., sycophancy or ignoring instructions), with prompts that elicit the behavior as evidence and XieX_{ie} again obtained by running the models. This is close in spirit to bias bounties (Globus-Harris et al., 2022), where external participants are rewarded for finding where a model underperforms; an evidence market adds a price on the comparison and pays most for failures that separate the models. (ii) Mathematics and code: for questions like "will this conjecture be proved by a given date?" or "does this paper's code reproduce its reported result?", the evidence is machine-checkable (formally verified lemmas, test cases, re-runs), so its correctness can be verified mechanically rather than by an LLM judge. (iii) LLM agents as traders: evidence markets with exogenous resolution can evaluate LLM agents on the evidence they find, not only on the accuracy of their forecasts, using questions that resolve after the agents' training cutoff. The pilot described above is an early example. (iv) Beyond ML, a classic case is disease forecasting: standard prediction markets forecasting influenza hospitalizations can only reward participants for their forecasts and not for their information behind said beliefs, but many traders with first-hand clinical information (e.g., health care workers) may not know how to express what they know as a calibrated probability. Nevertheless, such participants have valuable information bearing on the prediction; an evidence market rewards both.

References

  • T. Lin and Y. Chen. Generalized Principal-Agent Problem with a Learning Agent. ICLR 2025.
  • S. Hossain, E. Micha, Y. Chen, A. Procaccia. Strategic Classification with Externalities. ICLR 2025.
  • F. Bacchiocchi, M. Castiglioni, A. Marchesi, N. Gatti. Learning Optimal Contracts: How to Exploit Small Action Spaces. ICLR 2024.
  • M.-F. Balcan, M. Bernasconi, M. Castiglioni, et al. Nearly-Optimal Bandit Learning in Stackelberg Games with Side Information. ICLR 2026.
  • M. Hajiaghayi, S. Lahaie, K. Rezaei, S. Shin. Ad Auctions for LLMs via Retrieval Augmented Generation. NeurIPS 2024.
  • S. Hossain, Y. Chen. Equilibrium of Data Markets with Externality. ICML 2024.
  • I. Globus-Harris, M. Kearns, A. Roth. An Algorithmic Framework for Bias Bounties. FAccT 2022.

Comments on the paper

1 comment

We thank all reviewers for their thoughtful feedback. We are glad they found the work a novel extension of prediction markets with clear theoretical guarantees; well motivated, addressing a real gap by rewarding evidence alongside predictions, with a concrete mechanism and formal guarantees; and parsimonious and interpretable, with a simple connection between the reward for evidence and market uncertainty (R3).

Main points of our responses:

Our contribution is the mechanism and its guarantees. Truthful belief and full evidence submission form a dominant strategy equilibrium under exogenous resolution and an ε-SPNE under endogenous resolution, and worst-case loss is β0log⁡n\beta_0 \log n, identical to a plain LMSR. ICLR has a strong record of such theoretical work in game theory and mechanism design, and the mechanism targets a problem central to the community: obtaining fresh, discriminative LLM evaluation data without an external ground truth.

This mechanism holds for any non-negative, monotone quality function and any verifier. In that sense, the discriminative score and staked disputes are instantiations, not requirements. Errors in rr that preserve monotonicity change payments, not incentives, and the loss bound holds for any rr. Admission criteria tying evidence to a target task are the market designer’s choice.

Lastly, while there are many ways to motivate evidence submission, our approach funds evidence rewards within the standard LMSR subsidy, dynamically prices evidence based on the current uncertainty of the market, and dampens free-riding on public evidence.

Revisions to the paper:

  • A new appendix reporting a pilot of the evidence-gathering and verification components (R1, R2). Nineteen participants directing LLM agents submitted 815 pieces of evidence on 57 real-world questions; the LLM judge rejected 121 submissions and assigned positive quality to 466.
  • Theorem 5.1: following R3’s suggestion, reports under endogenous resolution are restricted to beliefs bounded below by pmin⁡p_{\min}. The truthful report remains admissible, and the rest of the proof is unchanged.
  • An explicit statement of the division of responsibility between the mechanism, whose incentive guarantees are domain-agnostic, and the market designer’s quality function and verification regime (R3).
  • New paragraphs in the Discussion (Section 6) on verification and evidence quality.

We hope these address the reviewers’ concerns and are happy to clarify further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.