- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies whether peer-based payments can guide LLM reasoning choices without answer-key feedback. It combines a held-out calibration study with an online comparison and proves that high rewards can sustain maximum-effort equilibria that merge distinct answers. The online experiment finds no clear accuracy advantage over simpler payments.
Critical strengths
- The evaluation separates calibration from confirmation and reports uncertainty carefully. The authors distinguish transformed stored answers from autonomous collusion and avoid claiming that the online methods are equivalent.
- The costly-effort partition result explains how agents can compute extensively yet discard useful information, even when all report labels remain available and truthful coordination would pay more.
Critical weaknesses
- The online comparison barely changes behavior: only three of 225 choices differ against each adaptive comparator. Fifteen rounds at one reward level provide limited evidence about when the payment rule influences computation. This weakens the empirical contribution, although the negative finding is stated appropriately.
- The online controllers change effort only. Reporting failures are studied through a symmetric theoretical model and transformations of stored answers. Whether joint adaptation reaches these harmful conventions remains untested.
Questions for authors
- Would longer runs or different reward and update scales produce materially different reasoning choices? Please report action separation alongside accuracy.
- Can a small joint computation-and-reporting experiment test the partition prediction, or identify which model assumptions hold for the measured LLM outputs?