- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether simpler interfaces and clearer task descriptions help LLM agents make better decisions. It studies auctions and school matching, where the correct strategy is known. The authors compare different interfaces and reasoning prompts across four model families. They also compare what agents say in their plans with what they actually do. Some interventions improve decisions without a similar change in the measured language indicators.
Critical strengths
I found the choice of setting useful. In many agent evaluations, it is hard to say why one decision is better than another. Here, the economic mechanisms provide a clear reference strategy. This makes the effects of interface and description changes easier to interpret.
The paper also goes beyond asking whether a prompt improves the final score. The comparison between stated plans and actual bids is interesting. For example, an intervention can reduce bid errors while agents still use similar language in their explanations. This is a useful finding for people who use agent explanations to judge whether an intervention works.
Another good part is that the results are not uniformly positive. Asking for more planning or reasoning about opponents sometimes makes decisions worse. The paper gives a more useful picture than simply recommending more reasoning for every task.
Critical weaknesses
The matching-interface result needs a clearer presentation. Figure 1 puts full-ranking and partial-ranking Kendall distances next to each other, but they do not measure the same thing. A zero in the iterative condition can come from having too little ranking information to score. The authors explain this and provide decision-level errors, but the figure still gives a stronger impression of improvement than the comparison supports. A common outcome measure would be much easier to read.
I am also less convinced by the measurement of verbal understanding. Most indicators use a fixed keyword dictionary, and agreement with the additional LLM judge is moderate. The safety-invariant feature is almost always absent in the main panel. With such a low starting rate, finding no change in this feature tells us relatively little. The paper already limits its claim to measured language, which is appropriate, but a small manually checked sample would make this part stronger.
Some statistical results also need caution. The two effective primary interventions have adjusted q-values of 0.068 when correction is applied across all interventions. Their bid effects are not significant in the four-model-cluster check either. This does not mean the effects are absent, but the evidence for generalizing across model families is weaker than the pooled results alone suggest.
Finally, Appendix C.8 changes how I read the practical contribution. Stronger models already make almost correct bids in the baseline, leaving little room for the main beneficial interventions. The frontier failures under alternative wording are interesting and deserve more attention in the main paper. They help explain where interface design still matters.
Questions for authors
Can the matching interfaces be compared using the same outcome measure, such as payoff loss relative to truthful play?
2 comments
Dear Reviewer id1i,
Thank you again for your comments.
Beyond the rebuttal, we have also uploaded a revised PDF that follows the suggestions you made.
At a glance:
-
Your question was whether the matching interfaces can be compared on the same outcome measure. Figure 1 now does this with assignment error (9.5% under standard DA vs. 3.6% under iterative DA), and the caption reports value lost relative to truthful reporting, which you asked about (0.22 vs. 0.12, on values between 40 and 90).
-
On the verbal measures: we've expanded the analysis to include two independent LLM judges (Claude Sonnet 5.5 and GPT-6.1 Sol), which labeled the 2,939 unique plans in the main cells from the plan text alone. They agree on 94% of labels (mean κ = 0.78), and the paper's results hold on their labels (Appendix C.3, Table 6). Table 7 gives sample plans with both judges' labels, so the classification can be checked by hand.
-
On the statistics: Table 1 now reports a within-model permutation test next to the original one (p = 0.012 for Payoff Tree, p = 0.025 for Payoff Safety).
-
On the frontier models: the failure under the menu wording is now in Section 5 of the main text, as you suggested.
If these changes resolve your concerns, would you consider updating your score? And if anything still looks unresolved, please tell us and we will address it before the discussion closes.
Thank you,
The authors
Thank you for the close reading and for recognizing the value of our experimental setting, behavioral analysis, and the findings. We also thank you for your thoughtful comments. We especially appreciated the suggestion of a payoff-based comparison for the matching results, which we have now adopted, and we were glad the comparison between stated plans and realized bids was useful to you.
Thank you for this suggestion. Kendall distance is the standard measure of how far a reported ranking departs from the truth, but it is undefined for iterative DA, where a student can be matched after a single query (hence why we reported the two interfaces separately). That being said, we agree a common outcome measure is much easier to read, and the revised Figure 1 also scores both interfaces by assignment error (the share of students whose final match differs from their match under truthful reporting) and by value lost relative to truthful reporting. The result holds:
Thank you for the comment. Our keyword features are simple matches that are usually right when they fire, which is why we used them. That being said, we agree a check strengthens this part, and we have added a semantic validation. Two LLM judges from different model families (Claude Sonnet 5.5 and GPT-6.1 Sol) labeled the 2,939 relevant plans. They agree on 94% of labels (mean κ = 0.78), give nearly identical treatment effects, and detect the language changes that other prompts induce (worst-case reasoning rises from 2% to 46%). On these labels, the effective prompts improve bids in every model while barely changing what the agents say: under Payoff Safety, valid safety reasons rise only in Claude and Gemini, not in GPT-4o or Gemma, whose bids improve most. Following your suggestion, we have also included a sample of labeled plans in the appendix so readers can check the classification by hand.
Thank you for raising this. The four-model check permutes treatment across models whose baseline errors differ fivefold, so it mostly measures differences between models rather than the effect of the prompt. Each model contributes one treated and two baseline runs. A permutation within model, which respects that design, gives p = 0.012 for Payoff Tree and 0.025 for Payoff Safety: in all four models, the treated run has the lowest error of that model's three runs, so no other assignment of treatment within models would have produced a stronger result. In matching, both prompts reduce misreporting in every model. We have updated Table 1 to report both tests.
We agree, and thank you for the suggestion; we have moved them into Section 5. As reported in Appendix C.8, under a menu restatement, Claude Sonnet 5 writes a formally correct derivation of the wrong mechanism's equilibrium and bids $8.70 away from value. Its explanation looks like understanding while its decision is wrong, which is why we evaluate decisions against a known benchmark.
Thank you again for the close reading; we believe your comments have helped us improve the paper.