- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ElicitBench tests group decisions requiring private facts. Co-Adapt uses alternating training for information elicitation. The authors report higher strict success on held-out domains.
Critical strengths
The work is comprehensive and includes extensive experiments.
Critical weaknesses
- The main contribution of this paper should be the proposed benchmark. I believe the proposed method "co-adapt" lacks of innovation because there is no new algorithm but a mixture of existing technologies. So I believe the correct "primary area" should be "datasets and benchmarks", rather than currently chosen "other topics in machine learning (i.e., none of the above)".
- I find the significance of the proposed elicitation problem limited by its incentive structure. All participant policies optimize the same training reward (Appendix C.2). Disclosure is rewarded, while free actions taken with unrevealed private facts are penalized (Appendix C.1). No private disclosure cost or individual benefit from secrecy is modeled. For example, withholding may be individually optimal if disclosure carries a private cost exceeding its expected benefit to the speaker, or if revealing a colleague's contribution would reduce the speaker's own bonus. The participants therefore have no conflict of interest over disclosure: pooling their decision-relevant facts provides a straightforward cooperative solution to the information-access problem. The framing does not adequately reflect this simplification. Lines 48–49 pose learning to elicit unobserved information as an open question; Section 3.1 emphasizes reasons people might not volunteer information. However, these narrative motives are not represented as conflicting incentives in the formal objective. Limited attention, communication, and reasoning may still make execution difficult, but the benchmark does not establish a substantive strategic disclosure challenge. I therefore find the motivation for this particular setting insufficiently convincing.
Questions for authors
None