- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes evidence markets, in which traders report both beliefs and supporting evidence. It extends the logarithmic market scoring rule by making its liquidity parameter decrease with the quality of accumulated evidence. The resulting payoff rewards informative evidence, with an entropy-dependent component. For exogenous resolution, the authors claim dominant-strategy truthful belief reporting and full evidence disclosure; for endogenous resolution, the outcome is sampled from a softmax of scores computed on submitted evidence, and the authors claim an approximate subgame-perfect equilibrium. The paper also gives a cost-function AMM representation, a bounded-loss argument, and practical sketches for verification and asynchronous execution. LLM evaluation is the principal motivating application. The evaluation is entirely theoretical, with no experiments or empirical validation.
Critical strengths
-
A relevant and appealing problem formulation. Incentivizing participants to disclose the evidence behind their forecasts could make prediction markets more interpretable and create reusable artifacts, including evaluation questions for LLMs. Treating evidence as an explicit economic input, alongside beliefs, is a worthwhile research direction.
-
A parsimonious mechanism with an interpretable payoff. Evidence-dependent liquidity yields a simple connection between the reward for evidence and market uncertainty. The bounded-loss calculation and cost-function AMM representation make the proposal concrete. In the exogenous setting, the basic full-disclosure incentive is particularly transparent.
Critical weaknesses
-
A substantial gap between the model and the proposed LLM-evaluation application. The model endows each trader with a fixed evidence set and analyzes selective disclosure; it does not incentivize the creation or acquisition of useful evidence. It also assumes a monotone, effectively verifiable quality measure. More importantly, endogenous resolution evaluates models on the evidence voluntarily submitted to the market. Full disclosure of these privately selected sets does not imply that the collected questions are representative of any target task distribution: an entirely valid but systematically selected set may reverse the ranking of models on that distribution. This is not merely an implementation detail, because the paper motivates endogenous resolution as a way to answer which model is better at a task. The authors should either identify assumptions connecting submitted evidence to the target evaluation objective, or narrow the claim to ranking models on the collected evidence.
-
The incremental technical contribution of evidence-dependent liquidity is insufficiently established. Many guarantees follow directly from familiar ingredients: properness of the logarithmic score, monotonicity of the evidence-quality function, telescoping payments, and softmax sensitivity. As a comparison point, a standard constant-liquidity LMSR plus a bounded, monotone cumulative evidence bonus already yields truthful belief reporting, full disclosure of pre-existing evidence, and bounded platform loss under exogenous resolution. The paper does not establish which economically meaningful property specifically requires evidence-dependent liquidity, rather than a separable evidence reward. The approximate endogenous incentive guarantee also relies on increasing softmax temperature, thereby making resolution less informative. A formal separation from the simple baseline or a quantitative incentive-versus-informativeness result would make the contribution significantly more compelling.
Questions for authors
-
For LLM evaluation, what is the target task distribution? What ensures that the strategically selected evidence is representative of it? If no such assumption is made, is the intended output only a ranking on the collected evidence?
-
Which guarantee specifically requires evidence-dependent liquidity? A conventional LMSR plus a bounded, monotone evidence bounty also supports truthful belief reporting and full disclosure under exogenous resolution. Can the authors show a formal separation or an additional property?
-
Theorem 5.1 uses a lower bound on the previous report seemingly derived from truthful reporting. Since SPNE covers off-path histories, how is this bound ensured for arbitrary admissible past reports? Is a truncated report space intended? I regard this as potentially fixable.
2 comments
Thank you for the detailed response. I appreciate the clarifications and the additional pilot study. The proposed restriction on the report space also appears to address my concern regarding Theorem 5.1.
However, my two main concerns remain.
First, while representativeness can reasonably be delegated to the market designer, full disclosure of existing evidence does not guarantee reliable model rankings under the target task distribution. The pilot study does not establish this connection.
Second, the incremental technical contribution of evidence-dependent liquidity remains unclear. The proposed advantages are interesting, but no formal separation from a constant-liquidity LMSR with a bounded, monotone evidence bonus is established. In fact, the same softmax sensitivity argument appears sufficient to obtain an approximate endogenous incentive guarantee for this simpler baseline under comparable assumptions.
Overall, I appreciate the authors' clarifications, but these concerns remain substantial. I therefore maintain my original rating of 2.
We thank the reviewer for the careful review and for recognizing the strengths of our work: the relevance of eliciting evidence alongside beliefs, and the parsimony and interpretability of the mechanism. The reviewer notes that evidence-dependent liquidity “yields a simple connection between the reward for evidence and market uncertainty.” That parsimony, and that connection, are precisely part of our answer to the question of incremental contribution below.
Representativeness of the collected evidence: This is a mechanism-design paper. Its contribution is a mechanism that lets traders convey why they trade, by incentivizing full disclosure of evidence alongside beliefs, and that can resolve from that evidence when no external resolution exists. It is agnostic to the quality function: the discriminative score in Appendix C is one instantiation, and our guarantees hold for any non-negative, monotone r (Definition 3.2)
Every assessment of whether one model is better than another relies on a selected dataset, whose relationship to the capability of interest depends on how it was constructed. This holds for static or dynamic benchmarks and is not specific to our setting. What our mechanism adds is that traders are incentivized to disclose all the evidence they hold, including evidence that cuts against their own position (Theorem 5.1).
Admission criteria connecting evidence to a target task are the market proposer’s design choice, encoded in r and its verification regime (Definition 3.2, Appendix C), e.g., relevance to a task specification, coverage across categories, or exclusion of near-duplicates. The mechanism’s incentive guarantees are domain agnostic. We make this division of responsibility explicit in the revision.
Contribution of evidence-dependent liquidity: We do not claim it is the only route to the exogenous guarantees; a constant-liquidity LMSR with a separate bonus can also deliver them. The question is what the design achieves beyond them: Parsimony within the same budget. As the reviewer notes, the designer specifies a single schedule β(⋅). Worst-case loss remains β(R0)logn,identical to a plain LMSR (Proposition 5.1): evidence rewards are funded within the liquidity subsidy and spent only as evidence arrives. At a fixed budget, a separable bonus adds a second parameter, lowering liquidity on every history, and thus reducing the sharpness of information aggregation .
Importantly, evidence in our setting is priced by remaining uncertainty. The expected evidence payoff is xH(qt) (Lemma 5.1, where H is the Shannon entropy): evidence earns most when the market is uncertain and nothing once it is settled. This follows directly from the single schedule, not from a separately designed weighting.
Lastly, our design dampens free-riding on public evidence. Suppose a trader submits evidence but does not trade, leaving the market price unchanged; this is the risk-free, evidence-only trade of Section 5.1. A later trader can read that evidence and move the price accordingly, collecting the belief-update reward for information they did not supply. Under constant liquidity, that later trader collects the full reward. Under our design, the submitted evidence has already lowered liquidity, so the later trader’s reward shrinks and part of the subsidy goes to the evidence supplier instead.
On the softmax temperature: A larger τ affects only the sharpness of the resolution lottery, not the information collected. Since full disclosure is incentivized, any function of the cumulative evidence (e.g., the argmax of scores) can be reported alongside the market independently of τ (Section 4). The trade-off is between lottery sharpness and the incentive bound, not between incentives and the informativeness of the collected evidence.
Lower bound in Theorem 5.1: We thank the reviewer and adopt the suggested truncated report space. Restricting reports under endogenous resolution to beliefs bounded below by pmin keeps the truthful report admissible, since every resolution belief already satisfies this bound (Lemma 4.1), so Proposition 5.1 and the rest of the proof are unchanged. The revisions are updated in the paper.