- Rating
- 1
- Confidence
- 1
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper investigates how to incentivize agents to provide decision-relevant predictions using joint scoring rules. The authors establish theoretical limitations for single-agent settings and propose mechanisms involving multiple agents. The approach is evaluated through experiments on MNIST and a code evaluation task using language models.
Critical strengths
The paper addresses an interesting problem and provides theoretical contributions regarding the limitations of single-agent mechanisms and the potential of joint scoring rules. The proposed framework is potentially relevant to decision-making applications.
Critical weaknesses
The paper is difficult to follow, and the presentation is not always well organized. For instance, Section 3 is titled "Results" but still introduces methodological contributions. More importantly, the motivation for using agents rather than simpler models or approaches is not sufficiently clear. The experimental evaluation is also limited. MNIST is a relatively simple dataset, and experiments on a single dataset provide insufficient evidence of the approach's generalizability. Furthermore, the dataset used in Section 4.2 is not clearly described.
Questions for authors
Could the authors better justify the need for agent-based mechanisms compared to simpler alternatives? How do the proposed mechanisms compare with such approaches? Could the authors clarify which dataset is used in Section 4.2 and provide stronger experimental evidence supporting the generalizability of their findings?
3 comments
I thank the reviewer for their rebuttal but my score remain unchanged.
Thank you for the fast response. Are there concerns you raised in the initial review that you feel are still unaddressed? Understanding which points, if any, you continue to find unclear or insufficient would help us determine whether further clarification is needed.
Thank you for taking the time to engage with our paper. Our aim is for the paper to be accessible to readers from a wide range of backgrounds, and your feedback is helpful for identifying where to provide further explanation.
The introduction has been re-written to include an example with a decision maker trying to maximize their expected utility, along with a visualization of the problem. It shows how truthful predictions can lead to suboptimal decisions, and why eliciting the right set of outcomes is challenging.
The paper's focus on agentic optimization allows us to establish formal guarantees about the reports that will be provided in equilibrium. This is akin to the proper scoring literature, which covers how honest and accurate predictions can be incentivized through the choice of reward. Our main contribution is defining reward structures that additionally incentivize predictions to be simple and decision-relevant, even when a principal lacks knowledge of what to elicit. In addition to the theoretical results, the precision of this approach allows it to outperform standard prompting methods on both simplicity and usefulness, as shown in our second experiment. If you have in mind other alternatives to agent-based mechanisms, we can provide further comparisons.
At your suggestion, we have renamed Section 3 from “Results” to “Mechanisms and Results”.
We agree that MNIST is a simple dataset, and we chose it for the first experiment as a proof-of-concept providing intuition, as well as an environment that would be easy for other researchers to build on. The second experiment uses the EvalPlus dataset, which contains examples of LLM generated attempts at solutions to 164 coding problems from the HumanEval dataset. We have moved additional details regarding that data from the appendix to the main body. While MNIST alone would not provide strong evidence of generalizability, the second experiment shows that the mechanism can be used to train and prompt LLMs on a task with practical applications.
We hope that these updates help clarify the motivation and contribution of our research, along with how the theory ties to experiments, and we ask that you please reconsider your rating of the paper. If you have specific concerns regarding the soundness of our results, we would welcome the opportunity to address them.