acceptodds
Under review as a conference paper at ICLR 2027

Engineering Simplicity: Simple Mechanism Interfaces Steer LLM Agents

Abstract

Can reasoning scaffolds and task descriptions help large language model (LLM) agents make better decisions? We study this question in a multi-agent testbed of auctions and matching, where economic theory specifies an optimal strategy and principles for making it easier to recognize. Across four model families, we vary the interface through which agents act and the descriptions accompanying their decisions. An ascending auction interface that makes truthful play obviously optimal substantially reduces bid errors. With the interface fixed, laying out payoff contingencies and stating why truth-telling is safe also improve choices, while prompts for forward planning or belief formation can worsen them. In auctions, these behavioral gains are not accompanied by improvements in measured verbal indicators of strategic understanding in agents' stated plans. Conversely, a scaffold that changes the targeted language leaves bids unchanged, and the interventions that change both worsen bids. Human-motivated theories of simplicity can therefore inform the design of decision environments for artificial agents. Their success should be evaluated through realized decisions alongside explanations: effective scaffolds and descriptions need not produce corresponding changes in measured verbal understanding.

Primary area:
learning theory
Keywords:
Language Model, Simplicity, Auction, matching

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_id1iOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether simpler interfaces and clearer task descriptions help LLM agents make better decisions. It studies auctions and school matching, where the correct strategy is known. The authors compare different interfaces and reasoning prompts across four model families. They also compare what agents say in their plans with what they actually do. Some interventions improve decisions without a similar change in the measured language indicators.

Critical strengths

I found the choice of setting useful. In many agent evaluations, it is hard to say why one decision is better than another. Here, the economic mechanisms provide a clear reference strategy. This makes the effects of interface and description changes easier to interpret.

The paper also goes beyond asking whether a prompt improves the final score. The comparison between stated plans and actual bids is interesting. For example, an intervention can reduce bid errors while agents still use similar language in their explanations. This is a useful finding for people who use agent explanations to judge whether an intervention works.

Another good part is that the results are not uniformly positive. Asking for more planning or reasoning about opponents sometimes makes decisions worse. The paper gives a more useful picture than simply recommending more reasoning for every task.

Critical weaknesses

The matching-interface result needs a clearer presentation. Figure 1 puts full-ranking and partial-ranking Kendall distances next to each other, but they do not measure the same thing. A zero in the iterative condition can come from having too little ranking information to score. The authors explain this and provide decision-level errors, but the figure still gives a stronger impression of improvement than the comparison supports. A common outcome measure would be much easier to read.

I am also less convinced by the measurement of verbal understanding. Most indicators use a fixed keyword dictionary, and agreement with the additional LLM judge is moderate. The safety-invariant feature is almost always absent in the main panel. With such a low starting rate, finding no change in this feature tells us relatively little. The paper already limits its claim to measured language, which is appropriate, but a small manually checked sample would make this part stronger.

Some statistical results also need caution. The two effective primary interventions have adjusted q-values of 0.068 when correction is applied across all interventions. Their bid effects are not significant in the four-model-cluster check either. This does not mean the effects are absent, but the evidence for generalizing across model families is weaker than the pooled results alone suggest.

Finally, Appendix C.8 changes how I read the practical contribution. Stronger models already make almost correct bids in the baseline, leaving little room for the main beneficial interventions. The frontier failures under alternative wording are interesting and deserve more attention in the main paper. They help explain where interface design still matters.

Questions for authors

Can the matching interfaces be compared using the same outcome measure, such as payoff loss relative to truthful play?

2 comments

Thank you for the close reading and for recognizing the value of our experimental setting, behavioral analysis, and the findings. We also thank you for your thoughtful comments. We especially appreciated the suggestion of a payoff-based comparison for the matching results, which we have now adopted, and we were glad the comparison between stated plans and realized bids was useful to you.

"The matching-interface result needs a clearer presentation."

Thank you for this suggestion. Kendall distance is the standard measure of how far a reported ranking departs from the truth, but it is undefined for iterative DA, where a student can be matched after a single query (hence why we reported the two interfaces separately). That being said, we agree a common outcome measure is much easier to read, and the revised Figure 1 also scores both interfaces by assignment error (the share of students whose final match differs from their match under truthful reporting) and by value lost relative to truthful reporting. The result holds:

ConditionAssignment error
Standard DA (complete ranking)9.5%
Iterative DA (OSP interface)3.6%
Standard DA + Payoff Tree3.8%
Standard DA + Rejection Safety0.5%

"I am also less convinced by the measurement of verbal understanding."

Thank you for the comment. Our keyword features are simple matches that are usually right when they fire, which is why we used them. That being said, we agree a check strengthens this part, and we have added a semantic validation. Two LLM judges from different model families (Claude Sonnet 5.5 and GPT-6.1 Sol) labeled the 2,939 relevant plans. They agree on 94% of labels (mean κ = 0.78), give nearly identical treatment effects, and detect the language changes that other prompts induce (worst-case reasoning rises from 2% to 46%). On these labels, the effective prompts improve bids in every model while barely changing what the agents say: under Payoff Safety, valid safety reasons rise only in Claude and Gemini, not in GPT-4o or Gemma, whose bids improve most. Following your suggestion, we have also included a sample of labeled plans in the appendix so readers can check the classification by hand.

"Some statistical results also need caution."

Thank you for raising this. The four-model check permutes treatment across models whose baseline errors differ fivefold, so it mostly measures differences between models rather than the effect of the prompt. Each model contributes one treated and two baseline runs. A permutation within model, which respects that design, gives p = 0.012 for Payoff Tree and 0.025 for Payoff Safety: in all four models, the treated run has the lowest error of that model's three runs, so no other assignment of treatment within models would have produced a stronger result. In matching, both prompts reduce misreporting in every model. We have updated Table 1 to report both tests.

"The frontier failures under alternative wording are interesting and deserve more attention in the main paper."

We agree, and thank you for the suggestion; we have moved them into Section 5. As reported in Appendix C.8, under a menu restatement, Claude Sonnet 5 writes a formally correct derivation of the wrong mechanism's equilibrium and bids $8.70 away from value. Its explanation looks like understanding while its decision is wrong, which is why we evaluate decisions against a known benchmark.

Thank you again for the close reading; we believe your comments have helped us improve the paper.

Dear Reviewer id1i,

Thank you again for your comments.

Beyond the rebuttal, we have also uploaded a revised PDF that follows the suggestions you made.

At a glance:

  1. Your question was whether the matching interfaces can be compared on the same outcome measure. Figure 1 now does this with assignment error (9.5% under standard DA vs. 3.6% under iterative DA), and the caption reports value lost relative to truthful reporting, which you asked about (0.22 vs. 0.12, on values between 40 and 90).

  2. On the verbal measures: we've expanded the analysis to include two independent LLM judges (Claude Sonnet 5.5 and GPT-6.1 Sol), which labeled the 2,939 unique plans in the main cells from the plan text alone. They agree on 94% of labels (mean κ = 0.78), and the paper's results hold on their labels (Appendix C.3, Table 6). Table 7 gives sample plans with both judges' labels, so the classification can be checked by hand.

  3. On the statistics: Table 1 now reports a within-model permutation test next to the original one (p = 0.012 for Payoff Tree, p = 0.025 for Payoff Safety).

  4. On the frontier models: the failure under the menu wording is now in Section 5 of the main text, as you suggested.

If these changes resolve your concerns, would you consider updating your score? And if anything still looks unresolved, please tell us and we will address it before the discussion closes.

Thank you,

The authors

Reviewer_nEhjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether simplicity principles from mechanism design can improve LLM agents’ decisions in second-price auctions and student-proposing deferred acceptance. The authors compare standard and obviously strategy-proof interfaces and evaluate several descriptions and reasoning scaffolds while holding the interface fixed. Across four main models, ascending auctions, payoff trees, and safety descriptions improve truthful play, whereas some planning and belief prompts worsen performance. The paper also analyzes short stated plans and reports that behavioral improvements do not consistently coincide with changes in selected verbal indicators.

Critical strengths
  1. The experimental settings provide explicit normative benchmarks. Dominant strategies in second-price auctions and student-proposing DA allow the authors to evaluate decisions independently of the quality of the accompanying explanations.

  2. The comparison between behavioral outcomes and verbal explanations addresses a useful evaluation question. Reporting both helpful and harmful interventions also provides practical evidence that requesting additional reasoning does not uniformly improve decisions.

Critical weaknesses
  1. The validity of the verbal measures is insufficiently established, weakening a central contribution of the paper. The intervention analysis relies on keyword and regular-expression features, but these features have not been validated against a human-annotated reference. Appendix C.3 reports an LLM-judge cross-check on 2,573 plans, with κ = 0.55 at the coarse classification level. This does not directly establish the validity or sensitivity of the individual features used to estimate intervention effects. The broader semantic safety flag is also not used to re-estimate those effects, and the proposed human validation has not been conducted.

There are concrete reasons to question measurement sensitivity. The strict safety feature fires on only 3 of 21,990 plans, and the dictionary does not handle negation, producing documented misclassifications in stronger models. Moreover, Table 1 describes the Payoff Tree indicator as rule rehearsal, while Table 6 identifies it as second_price_mention, which Table 3 distinguishes from payment_rule_correct. Merely mentioning the mechanism and correctly explaining its payment rule are materially different measures.

I appreciate the authors’ qualification that they do not observe internal understanding. Nevertheless, even the narrower conclusion about behavioral and verbal changes requires sufficiently sensitive and consistent measures of verbal content. The current evidence does not adequately distinguish a substantive dissociation from limitations of the measurement procedure. Human annotation is not inherently required, but a credible semantic validation of the relevant measures is necessary here.

  1. The remaining behavioral evidence does not establish a sufficiently strong incremental contribution. As the paper acknowledges, the auction advantage of OSP interfaces builds on prior work. The additional contribution therefore depends substantially on the specific prompt interventions and the accompanying verbal analysis. However, the Payoff Safety and Payoff Tree effects have q = 0.068 under the full intervention correction and are not supported by the four-model-cluster check. These results do not establish that the effects are absent, particularly given the limited power of a four-model test, but they make the strength of the evidence sensitive to the inferential specification.

The scope of the gains is also limited by the paper’s own stronger-model extension: Appendix C.8 reports a pooled baseline error of approximately $0.03, leaving essentially no improvement from the main helpful interventions. This boundary condition matters for the practical significance of the proposed design principles. Taken together with the unresolved validity of the verbal analysis, the current results provide useful observations but insufficient evidence for the paper’s overall contribution.

Questions for authors
  1. Beyond the specific prompt designs tested, what new insight does this paper establish over prior work on OSP and mechanism descriptions? What evidence shows that this insight generalizes beyond the particular wording used?

  2. Does the finding that bids improve without corresponding changes in stated understanding hold when plans are evaluated for their meaning? How do you rule out the possibility that the keyword-based measures simply fail to detect changes in the explanations?

3 comments

Thank you for the close reading and acknowledging our experimental setting and analysis for both actions and stated reasoning! Also thanks for your detailed comments. We enjoyed thinking through them, and they led to new analyses that we think make the paper considerably stronger.

"The validity of the verbal measures is insufficiently established ... The broader semantic safety flag is also not used to re-estimate those effects."

Our keyword features are simple matches: they miss some paraphrases, but when they fire they are usually right, which is why we used them for the intervention analysis. That being said, we agree that a semantic validation makes this stronger, and we have now run one. Two froniter LLM judges from different model families (Claude Sonnet 5.5 and GPT-6.1 Sol), shown only the plan text and blind to cell, model, and bid, labeled the 2,939 unique plans in the relevant cells. They agree on 94% of labels (mean κ = 0.78) and on 99% of intended bid directions, and give nearly identical treatment effects (for example, valid safety reasons under Payoff Safety rise from 2.5% to 7.5% for one judge and from 2.5% to 6.4% for the other). The judges also confirm that the keyword features are usually right when they fire, while catching more paraphrases (stated shading in 81% of plans, against 54% for the keywords). Finally, they pass positive controls: under the prompts that target them, they detect worst-case reasoning rising from 2% to 46%, dominance reasoning from 7% to 59%, and opponent modeling from 76% to about 90%, none of which improves bids.

Both judges confirm the central finding: the prompts that improve bids barely change what the agents say in their explanations. Under Payoff Safety, bid error falls 39%, with every model improving, but only 7.5% of treated plans give a valid reason for why truthful bidding is safe, and the two models whose bids improve most (GPT-4o and Gemma) show no increase in such statements. Under Payoff Tree, bid error falls 60% while 83% of plans still say to bid below value. We have revised the text to state the result this way, and thank you for catching the Table 1 label, which we have corrected.

"The Payoff Safety and Payoff Tree effects have q = 0.068 ... and are not supported by the four-model-cluster check."

The four-model check permutes treatment across models whose baseline errors differ fivefold, so it mostly measures differences between models rather than the effect of the prompt. Each model contributes one treated and two baseline runs. A permutation within model, which respects that design, gives p = 0.012 for Payoff Tree and 0.025 for Payoff Safety: in all four models, the treated run has the lowest error of that model's three runs, so no other assignment of treatment within models would have produced a stronger result. In matching, both prompts reduce misreporting in every model. We have updated Table 1 to report both tests.

"Appendix C.8 reports a pooled baseline error of approximately $0.03, leaving essentially no improvement from the main helpful interventions. This boundary condition matters for the practical significance of the proposed design principles."

We agree that frontier models already bid truthfully in this auction, so the helpful descriptions have nothing left to fix there, which is why we report it as a boundary condition. That being said, the design principles still matter at the frontier, because frontier models remain sensitive to how a rule is described. As reported in Appendix C.8, restating the same second-price rule as a menu raises Claude Sonnet 5's error from about $0 to $8.70 (it mistakes the auction for a first-price one), while the descriptions the paper recommends, Payoff Safety and Payoff Tree, leave frontier play at the truthful baseline. For models that already play well, the principles tell a designer which descriptions are safe to use and which can break play. We have moved this result into Section 5.

"Beyond the specific prompt designs tested, what new insight does this paper establish over prior work on OSP and mechanism descriptions? What evidence shows that this insight generalizes beyond the particular wording used?"

Thank you for the question. Fundamentally, the most interesting finding is that a theory built to describe which mechanisms are simple for humans also predicts which are simple for language models, across two different mechanisms, and that it yields design advice that runs against common practice.

More practically, our work is motivated by the rising use of LLM-based agents to bid, shop, and act on behalf of people in markets that were designed for human participants (e.g. Meta Muse, OpenAI Dot and Grok Bot). Auction and matching rules have been designed with human actors in mind, so we need to know how these agents behave under the same rules. Our results show that LLM agents are a different kind of participant from each of the benchmarks that market design has relied on. They are not the rational agent of economic theory: truthful play is a dominant strategy in each mechanism studied, yet the four main models deviate from truthfulness, and two descriptions of the same second-price rule produce different bids (restating the rule as a menu raises Claude Sonnet 5's error from about $0 to $8.70). And while the theory of simple mechanisms predicts which mechanisms are hard for both, LLM agents are not human bidders either. Human subjects in second-price auctions tend to bid above their value (Kagel and Levin, 1993), while these models bid below it, carrying a first-price heuristic into a second-price auction: under Payoff Tree, 83% of plans still say to bid below value. And when their bids improve, their stated reasons do not: under Payoff Safety, bid error falls 39%, but only 7.5% of treated plans give a valid reason that truthful bidding is safe.

So, for mechinism designers facing more LLM agents in the system/platform, the paper gives three high-level design recommendations. 1) Use a sequential interface where one exists: the ascending clock lowers bid deviations in all four model families. 2) Describe what each action leads to, or why truthful play is safe. 3) Do not ask the agent to plan ahead or to model its opponents, which does not help and can make play worse. These recommendations do not depend on a particular wording: in DA, four differently phrased statements of the safety property all lower assignment error (to 0.5%–4.9%, from 9.5%), while a menu that restates only the mechanics raises it to 16.2%.

We will revise the paper to add this discussion. Thank you again for the close reading; we believe your comments have helped us improve the paper.

Dear Reviewer nEhj,

Thank you again for your comments.

Beyond the rebuttal, we have also uploaded a revised PDF that addresses the helpful comments you made. Could you take another look and tell us whether it changes your assessment?

At a glance:

  1. Your most pressing comment was on our verbal measures and the plan analysis. We’ve since expanded the analysis here to include two independent LLM judges (Claude Sonnet 5.5 and GPT-6.1 Sol), which labeled the 2,939 unique plans in the main cells from the plan text alone. They agree on 94% of labels (mean κ = 0.78), and the paper’s results hold according to their labels. Under Payoff Safety, bid error falls 39% while valid safety reasons rise only from 2.5% to 7.5% (Section 5, Appendix C.3, Table 6). Table 7 shows sample plans with both judges' labels. All implementation details are included in the text and appendix, and we’ll push the code along with the open-source repo for camera-ready.

  2. On the Payoff Tree label: Table 1 and the full table (now Table 8) both score it on correct statements of the payment rule.

  3. On the statistics: Table 1 now reports a within-model permutation test next to the original one (p = 0.012 for Payoff Tree, p = 0.025 for Payoff Safety).

  4. On your question about what is new and whether it depends on wording: we rewrote Section 6 to reflect the language in our rebuttal more directly, and also added Table 11 to summarize the point. In particular, how we phrase the “safety” property doesn’t seem to matter much — four differently phrased statements of the safety property all lower assignment error in DA (to 0.5%–4.9%, from 9.5%), and a menu that restates only the mechanics raises it to 16.2%. The frontier-model result is also now included in Section 5 of the main text.

If anything still looks unresolved, please tell us and we will address it before the discussion closes.

Thank you,

The authors

Reviewer_uyF9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether simpler interfaces and clearer instructions help LLMs make better economic decisions. It studies two settings: second-price auction and school matching. The authors compare submitting a single bid with making repeated stay-or-exit decisions as the auction price rises. For matching, they compare submitting a complete ranking with answering questions about individual schools, under restricted school priorities. They also change the written instructions: “Payoff Tree” explains possible consequences, while “Payoff Safety” and “Rejection Safety” explain why truthful reporting is safe. Other prompts ask models to plan ahead or reason about opponents.

Experiments cover four main models. The paper measures both their decisions and the short explanations they write before bidding, asking whether better choices come with better explanations.

Critical strengths

Economic theory gives the experiments a clear benchmark. The authors can measure how far an action departs from truthful behavior.

Critical weaknesses

Two main concerns.

The two matching interfaces are not evaluated on a common scale. A complete ranking is scored by how many school pairs are ordered incorrectly. Sequential responses are scored by incorrect yes/no answers or final choices. Those percentages measure different things. Moreover, a response containing only one ranked school receives zero ranking error because there is no pair to compare.

The planning experiment changes the instructions as well as asking for more planning. The original explanation includes tentative acceptance and schools’ priority rules. The one-step planning treatment replaces it with a shorter explanation that omits those details, then asks the model to think ahead. Its worse performance could come from the changed explanation, the planning request, or both. A version without planning guidance is listed, but the paper does not report a comparison that holds the explanation fixed and adds only the planning request. Without that comparison, the result does not isolate the effect of extra planning.

Questions for authors

Please see my concerns in "Critical Weaknesses" above.

3 comments

Thank you for the close reading and the helpful comments and acknowledging our experiment design and theory grounding! Both your points are well taken, and we address each below with new analyses.

"The two matching interfaces are not evaluated on a common scale."

Kendall distance is the standard measure of how far a reported ranking departs from the truth, but it is undefined for iterative DA, where a student can be matched after a single query (hence why we reported the two interfaces separately). That being said, we agree a common outcome measure would be helpful, and the revised Figure 1 also scores both interfaces by assignment error: the share of students whose final match differs from their match under truthful reporting. The result holds:

ConditionAssignment error
Standard DA (complete ranking)9.5%
Iterative DA (OSP interface)3.6%
Standard DA + Payoff Tree3.8%
Standard DA + Rejection Safety0.5%

"The planning experiment changes the instructions as well as asking for more planning."

Thank you for the careful reading. You are right that the one-step planning prompt, in being more concise, drops the parts of the DA description that explain tentative acceptance and school priorities, so it changes the explanation as well as adding the planning request. To address this, we ran an additional variant that keeps the full DA description word for word and adds only the planning request:

DA promptAssignment errorCould gain by reporting truthfully
Full description (baseline)9.5%4.0%
Full description + planning request (new)12.6%5.4%
Short description + planning request (paper)21.0%9.6%

With the description held fixed, the planning request still raises assignment error, and it raises the share of students who could have done better by reporting truthfully (holding others' reports fixed), so it worsens the agents' own reports and not just the final matches. However, the harm is more modest than under the original concise prompt, so part of the original effect also comes from the shorter description. We have revised Section 4 to report both.

Thank you again for the comments; they were constructive and helped us improve the paper. We will also update the manuscript with details.

Dear Reviewer uyF9,

Thank you again for your comments.

Beyond the rebuttal, we have also uploaded a revised PDF that addresses both of the points you raised.

At a glance:

  1. On the two matching interfaces not being scored on a common scale: Figure 1 now scores both on assignment error, the share of students whose final match differs from their match under truthful reporting. It's 9.5% under standard DA and 3.6% under iterative DA, and the figure shows it for each model (the numbers are in Table 10).

  2. On the planning experiment changing the description as well as adding the planning request: we ran the comparison you asked for, which keeps the full DA description and adds only the planning request (Appendix E.2). Assignment error rises from 9.5% to 12.6%, and the share of students who could gain by reporting truthfully rises from 4.0% to 5.4%. The shorter one-step prompt gives 21.0%, so part of the original effect did come from the shorter description. Pairwise ranking error is essentially unchanged (4.2% vs. 3.9%), and we report that too.

If these changes resolve your two concerns, would you consider updating your score? And if anything still looks unresolved, please tell us and we will address it before the discussion closes.

Thank you,

The authors

Thanks for the additional experiments. They addressed my concerns. I will maintain my score and still lean towards acceptance.

Comments on the paper

1 comment
Area_Chair_HmcBOpenReview ↗

Reminder: Reviewer Engagement During the Discussion Period

Dear Reviewers,

As the discussion period is drawing to a close, could you please review the authors' rebuttal and provide your feedback at your earliest convenience?

Active engagement in discussions with the authors is an essential part of the peer-review process and plays an important role in ensuring the success of the conference. Your participation would be greatly appreciated.

Thank you for your time and contributions.

Best regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.