acceptodds
Under review as a conference paper at ICLR 2027

When LLMs Bid Rationally, and When They Don't: A Traceable Capability Split in Sealed-Bid Auctions

Abstract

Large language models (LLMs) are increasingly proposed as autonomous economic agents, but whether they comply with rational, incentive-compatible strategies in a canonical mechanism is rarely tested against closed-form ground truth. We study this in sealed-bid auctions, testing four LLMs against second- and first-price mechanisms. We find a clean capability split: two models, Gemini 3.5 Flash-Lite and gpt-oss-120b, are statistically indistinguishable from exact truthful bidding in our primary second-price test, contrasting sharply with the well-documented human tendency to overbid relative to the risk-neutral Nash equilibrium. The other two fail in mirror-image ways: Llama 3.1 8B overbids and, as bidder groups grow, degrades to bids indistinguishable from uniform-random noise; Qwen2.5-7B underbids by a comparable magnitude. Both trace to a symmetric declarative-knowledge error: each model states one mechanism's optimal rule correctly but carries it, incorrectly, into the other, giving a mechanistic account of both failures, not just a behavioral one. This double dissociation also produces a striking aggregation artifact: naively pooling all four models in our primary truthfulness test reverses its sign and significance entirely, since the two large, opposite-signed per-model effects nearly cancel under pooling, a cautionary result for any LLM-agent benchmark that aggregates across models rather than reporting per-model effects. We also test Regret-Conditioned Prompting (RCP), an inference-time feedback loop conditioning a model on its realized counterfactual regret each round; fit separately per mechanism with an expanded replicate sample, it produces a first-price-specific correction on our weakest model that survives Holm-Bonferroni correction (), though a cluster-robust check places it at the edge of conventional significance (); a matched follow-up on our underbidding model finds no comparable effect, so RCP's benefit is model- and mechanism-specific rather than general. Code and data will be released upon publication.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, game theory, sealed-bid auctions, mechanism design, incentive compatibility, quantal response equilibrium, bounded rationality, LLM agents, benchmark and evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_V9suOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tests whether four language models follow theoretically optimal strategies in first- and second-price auctions. Each trial places one model against synthetic opponents using known equilibrium strategies. The primary experiments use uniformly distributed private values and two or five bidders, with additional checks using larger groups and a non-uniform distribution. The authors compare bids with theoretical targets, elicit general bidding rules in separate contexts, and fit descriptive models of bidding behavior.

Two models closely follow the primary theoretical targets, while Qwen systematically underbids in second-price auctions and Llama makes occasional large overbids. The authors relate these differences to stated bidding rules and show how opposite-signed deviations cancel when pooled. They also compare regret-conditioned feedback with simpler controls. Reminding models of the optimal rule is more effective than regret-conditioned feedback in the reported first-price experiments. Extensive sensitivity checks qualify the strength of the statistical and explanatory conclusions.

Critical strengths
  1. The auction environment provides explicit theoretical targets and interpretable bidding errors. Known opponent strategies isolate the focal model’s behavior, supporting a clear and reproducible diagnostic evaluation.

  2. The paper transparently reports sensitivity checks and negative results. Per-model effects and the stronger performance of the simple theorem reminder offer useful lessons for evaluating and correcting bidding behavior.

Critical weaknesses
  1. The primary tests overstate the strength of evidence. Near-identical bids on shared valuation draws are treated as independent, and several analysis choices changed after results were observed. Accounting for dependence removes significance for Llama’s mean overbidding and weakens the feedback result. Although disclosed, these issues limit confidence in the headline tests. Dependence-aware analyses should lead, with adaptive findings consistently treated as exploratory.

  2. The explanation through stated rules is weakly established. Each condition has only one separately elicited rule statement. First-price declarative correctness requires the direction of shading, whereas behavioral correctness requires numerical accuracy. The authors acknowledge that requiring the exact equilibrium rule eliminates the crossed pattern. This limits the claimed knowledge–behavior distinction; comparable criteria or a narrower descriptive interpretation are needed.

Questions for authors
  1. Which central conclusions remain supported when the reported dependence-aware checks are primary and post hoc choices are treated as exploratory?

  2. Under comparable correctness criteria for stated rules and bids, what evidence remains for the proposed knowledge–behavior distinction?

Reviewer_XPAcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether LLMs bid optimally in sealed-bid auctions, where the optimal bid has a closed-form solution: bid one's value in a second-price auction, and a fixed fraction of one's value in a first-price auction. Each LLM bids against simulated opponents that follow the optimal strategy.

The paper claims the following findings and contributions: Gemini 3.5 Flash-Lite and gpt-oss-120b bid optimally, while two smaller models do not: Qwen2.5-7B consistently underbids, and Llama 3.1 8B bids erratically. This paper offers an explanation of why Qwen and Llama fail. The paper also gives RCP which tells the model each round how much it lost by not bidding optimally.

Critical strengths

This environment offers an exact measurement protocol, we know the exact ground truth number.

Critical weaknesses

The evidence does not appear to support the paper's conclusions. For example, RCP ultimately seems to show no benefit. The findings also cannot be generalized beyond the 4 configurations tested. The paper does not explain why the models differ, and it does not show whether its findings hold for other models or prompts.

Questions for authors
  1. What is the Input of the model(prompts, setup, ...etc)? Was the model told how the opponents bid and the number n? What are the results when the thinking mode is on(or let the model think)?
  2. Does the divergence persist within the same model family(for example, larger models in Qwen family)?
  3. Need more experiments and analysis of model outputs. the results seem to be fragile.
Reviewer_x5NkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates whether LLM bidding behavior conforms to game-theoretic predictions in first- and second-price sealed-bid auctions. It evaluates four models against synthetic opponents following theoretically prescribed strategies, compares stated bidding rules with actual behavior, and examines whether theorem reminders or repeated counterfactual feedback can improve bidding decisions. The results reveal substantial model heterogeneity: two models closely follow theoretical benchmarks, while the others exhibit distinct deviations, including systematic underbidding and noisy or excessive overbidding. The intervention experiments suggest that explicit theorem reminders are more effective than regret-conditioned feedback at correcting first-price overbidding in the weakest model.

Critical strengths
  1. A controlled setting with explicit theoretical benchmarks. Using synthetic equilibrium opponents allows bidding deviations to be measured against well-defined predictions without confounding them with another LLM’s changing strategy. This makes the observed differences across models interpretable and provides a useful diagnostic of economic reasoning.

  2. Interventions help distinguish knowledge access from behavioral adaptation. Comparing theorem reminders with counterfactual feedback goes beyond documenting bidding errors and tests concrete ways to correct them. The near-elimination of Llama's first-price overbidding under theorem reminders, compared with weak feedback effects, is an informative result about the conditions under which these models implement equilibrium strategies.

Critical weaknesses
  1. The primary significance tests overstate the effective sample size. The bid-elicitation stages of protocols A and B use identical content and shared valuation draws, producing identical bids in 93–100% of cases, yet both enter the primary analysis as separate observations. This materially affects the conclusions: Llama’s reported overbidding result changes from an unadjusted (p=.00798) (Holm-adjusted (p=.0319)) to (p=.060) after collapsing duplicates, while the corresponding Wilcoxon test gives (p=.424). Although the paper acknowledges this issue and reports sensitivity checks, the primary significance claim remains sensitive to these dependencies.

  2. Data-dependent design changes weaken the confirmatory interpretation. The paper documents adding Qwen after inspecting other results, revising hypothesis analyses, and increasing feedback-experiment replication after an initial (p=.084). These disclosures are valuable, but conventional multiplicity correction does not account for such data-dependent decisions, and the feedback result is itself sensitive to dependence-aware inference ((p=.053) with clustering versus (p=.00158) without).

Questions for authors
  1. Can you report the main hypothesis tests using the valuation draw within each experimental condition as the independent unit, accounting for dependence across prompt variants and shared draws?

  2. Given the documented design changes, which hypotheses, model choices, stopping rules, and feedback-effect specifications remained fixed before observing the relevant outcomes? Is there an untouched evaluation set or independent replication supporting the findings affected by subsequent changes, especially the reported benefit of counterfactual feedback?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.