acceptodds
Withdrawn from ICLR 2027

LLM BIDDERS PRESERVE THE MECHANISM-LEVEL ORDERINGS OF HUMAN BIDDERS

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Training on vast amounts of human-generated data has motivated growing interest in using large language models (LLMs) to simulate human behavior. We ask which features of human behavior general-purpose models preserve when used out of the box in auctions, where multiple bidders interact under explicit rules and incentives. We evaluate five LLMs across seven laboratory settings against human benchmarks reconstructed from published experiments, with uncertainty bands for the private-value comparisons. Our main focus is on three large models without extended test-time reasoning: GPT-4o, Claude 3.5 Haiku, and Gemini 2.0 Flash. LLM and human deviations from theory differ in magnitude and often in direction: humans overbid in second-price auctions, whereas most models that deviate underbid. Surprisingly, without task-specific fine-tuning or calibration to human bids, the three non-reasoning large models robustly preserve key orderings of auction formats by deviation from theory. First-price auctions are harder than second-price, and ascending clocks reduce deviations relative to sealed bids wherever data are adequate. Kendall's between the human and GPT-4o difficulty rankings is and positive in every joint bootstrap draw. The reasoning model bids almost at equilibrium in the observed private-value settings, leaving little variation in errors to compare; the small model's large errors yield an inverted ranking. All five models nevertheless reproduce the stronger first-price winner's curse. Clock framing improves bidding for two of the three non-reasoning large models, and GPT-4o recovers the ordering of last-minute bidding across closing rules in an eBay-style marketplace.

Primary area:
learning theory
Keywords:
Language Model, Auction, Bidding, Experiment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_RvRoOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether LLM bidders reproduce the ordering of auction formats observed in human experiments, even when their bids differ from human bids. It compares five models across seven auction settings using human benchmarks reconstructed from earlier studies. The main finding is that some models preserve several orderings despite differences in the size and direction of their deviations from theory.

Critical strengths

The paper addresses an interesting question: whether LLMs can help compare auction formats even when they do not accurately reproduce human bids. Establishing this would give researchers a useful, more limited alternative to treating LLMs as substitutes for human participants.

Critical weaknesses

The submission shows insufficient care in editing and proofreading. The main text appears to have been carelessly edited before submission, possibly due to AI assistance. For example, the OpenReview abstract contains strikethrough text. More seriously, the introduction claims that clock framing increases truthful bidding for Claude, while Table 4 shows that the share of bids close to the true value decreases. A lower average error does not mean more truthful bidding...

Important details are left to the appendices, and the main text does not clearly and consistently describe the measurements. In the common-value settings, it is unclear how the descriptions of errors in bids relate to the descriptions of errors in profits. Because the main claim depends on comparing deviations across auction formats, this affects how the results are interpreted.

These problems go beyond writing quality. They make it difficult to trust the connection between the measurements, results, and conclusions.

Questions for authors

How does the claim about increased truthful bidding for Claude agree with Table 4? What exactly is measured in each auction setting, and are those measurements comparable?

Reviewer_fSr6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates five LLMs across seven auction settings using human benchmarks reconstructed from published experiments. It distinguishes agreement in the size of bidding errors, their direction and the ordering of auction formats by deviation from theory. Three large models without extended reasoning preserve several human orderings even though their actual bids differ. First-price auctions produce larger deviations than second-price auctions. Ascending clocks reduce deviations where data are adequate. Further experiments examine the winner's curse, prompt framing and last-minute bidding under different marketplace closing rules.

Critical strengths

[+] Separating bid levels, direction, and ordering makes clear what the simulations reproduce and where they fail.

[+] Five models and several auction formats show how behavioral agreement varies across settings.

[+] Reconstruction bands and temperature checks test sensitivity to benchmark uncertainty and generation settings.

Critical weaknesses

[-] Section 2.1 and Appendix B.1 give different reconstruction rules. Common-value deviations also switch between bid-based and profit-based definitions. It is unclear how the reported tables were generated.

[-] Some clock cells contain only two usable bids, without attempt or exclusion counts. It is difficult to assess how missing observations affect the comparisons.

[-] Bid-level bootstrapping ignores dependence within affiliated-value and interactive auctions. The intervals should be checked by resampling whole auctions.

Questions for authors
  1. Which reconstruction and common-value metric produced the tables? Do the orderings survive matching human bidder counts?
  2. How many clock auctions were attempted, completed, and excluded? How were procedural failures handled?
  3. Do the concordance intervals and framing results hold under auction-level resampling?
Reviewer_kYs4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether LLMs can reproduce human bidding patterns across auction mechanisms without task-specific fine-tuning or calibration to human data. The authors compare five models against human benchmarks reconstructed from summary statistics in published experiments. They assess three aspects of behavioral similarity: the magnitude and direction of deviations from equilibrium, and the relative difficulty of different mechanisms. The main finding is that some large non-reasoning models reproduce several human difficulty orderings despite differing from humans in the magnitude and direction of their errors. The authors suggest using these models to screen mechanism designs before conducting human experiments.

Critical strengths
  1. The distinction between matching the size and direction of human errors and matching the ordering of mechanisms is useful. The results suggest that LLMs may help identify some design changes worth testing with human participants.
  2. The study covers several auction formats and model families. The proposed benchmark and planned code release could support further work on LLMs as models of behavior in strategic settings.
Critical weaknesses
  1. Table 5 on page 16 states that the mixture weights come directly from frequencies reported in the original studies. However, the value (p_{\mathrm{eq}}=0.75) for the five-bidder first-price auction attributed to Kagel and Levin (1993) matches neither the 1.7% equilibrium-bidding frequency nor the 7.5% value-bidding frequency in Table 2 on page 873 of the original paper. The source of this value and any transformations applied to the original statistics need to be explained.

  2. Two references have incorrect author information. AuctionNet (arXiv:2412.10798) is by Kefan Su et al., not Zhiyu Chen et al.; InfoBid (arXiv:2503.22726) is by Yue Yin alone, not Yue Zhang et al. These entries disagree with the original papers and official records, and the checked version histories do not explain the differences. The errors raise concerns about how carefully the references were checked. The authors should correct these entries and review the remaining citations.

  3. In Table 4, Claude’s SMAD falls from 8.1% to 7.1%, but the share of bids close to the true valuation falls from 28% to 7%. Lower average error therefore should not be described as more truthful bidding. For Gemini, SMAD rises from 5.0% to 9.3%, while the share of bids close to the true valuation falls from 62% to 37%. Gemini’s better baseline performance alone does not explain why the intervention makes its behavior worse, nor does it establish that this reversal was theoretically expected.

  4. Most of the evaluation concerns mechanism orderings already documented in the literature. The paper does not provide independent human validation for new mechanisms or interventions outside those used to construct the benchmark. Reproducing known orderings is an interesting result, but it provides limited evidence that LLMs can reliably screen designs whose effects on humans are not yet known.

Questions for authors
  1. Could the authors provide a source table for each human benchmark, listing the original page and table, experimental treatment, reported statistics, transformation formulas, and reconstruction code? What explains the differences between Table 5 and the original Kagel–Levin paper, and do the main orderings hold after these differences are resolved?

  2. Is there direct evidence from human experiments for the comparison between first-price and second-price sealed-bid common-value auctions? How were the two human common-value benchmark values in Table 1 calculated?

  3. Several LLM clock-auction cells contain very few usable bids, in some cases only two. Could the authors collect enough observations to meet their stated threshold of (n \geq 30) and check whether the ordering conclusions still hold?

Comments on the paper

1 comment

We decided to withdraw this submission. We thank the reviewers for their comments and will address them in a revised version of the paper. But we would like to raise two disagreements with the reviews .

Reviewer RvRo pointed to a strikethrough left in the OpenReview abstract as a sign of careless editing and speculated that the paper was written with AI assistance. The strikethrough was a proofreading mistake on our part. We do not think it should weigh in the evaluation of the paper, and the speculation about how the manuscript was prepared is unsupported.

The human benchmarks come from five published auction experiments: Kagel and Levin (1993) for first- and second-price sealed-bid auctions with independent private values; Li (2017) for sealed second-price and open ascending clock auctions with affiliated private values; Breitmoser and Schweighofer-Kodritsch (2022) for the ascending clock without dropout information; Kagel and Levin (1986) for first-price common-value auctions; and Levin, Kagel, and Richard (1996) for English common-value auctions. The appendix lists each source study and explains how we extracted the reported statistics and reconstructed the benchmarks from them. Questions about specific values, such as the mixture weight attributed to Kagel and Levin, are fair, and the revision will answer them precisely. These questions are about how we used existing evidence. The request for a new human experiment is a different question: it asks whether the findings hold in an auction design that has not yet been tested with human subjects. We would have liked the reviews to keep the two apart.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.