- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tests whether four language models follow theoretically optimal strategies in first- and second-price auctions. Each trial places one model against synthetic opponents using known equilibrium strategies. The primary experiments use uniformly distributed private values and two or five bidders, with additional checks using larger groups and a non-uniform distribution. The authors compare bids with theoretical targets, elicit general bidding rules in separate contexts, and fit descriptive models of bidding behavior.
Two models closely follow the primary theoretical targets, while Qwen systematically underbids in second-price auctions and Llama makes occasional large overbids. The authors relate these differences to stated bidding rules and show how opposite-signed deviations cancel when pooled. They also compare regret-conditioned feedback with simpler controls. Reminding models of the optimal rule is more effective than regret-conditioned feedback in the reported first-price experiments. Extensive sensitivity checks qualify the strength of the statistical and explanatory conclusions.
Critical strengths
-
The auction environment provides explicit theoretical targets and interpretable bidding errors. Known opponent strategies isolate the focal model’s behavior, supporting a clear and reproducible diagnostic evaluation.
-
The paper transparently reports sensitivity checks and negative results. Per-model effects and the stronger performance of the simple theorem reminder offer useful lessons for evaluating and correcting bidding behavior.
Critical weaknesses
-
The primary tests overstate the strength of evidence. Near-identical bids on shared valuation draws are treated as independent, and several analysis choices changed after results were observed. Accounting for dependence removes significance for Llama’s mean overbidding and weakens the feedback result. Although disclosed, these issues limit confidence in the headline tests. Dependence-aware analyses should lead, with adaptive findings consistently treated as exploratory.
-
The explanation through stated rules is weakly established. Each condition has only one separately elicited rule statement. First-price declarative correctness requires the direction of shading, whereas behavioral correctness requires numerical accuracy. The authors acknowledge that requiring the exact equilibrium rule eliminates the crossed pattern. This limits the claimed knowledge–behavior distinction; comparable criteria or a narrower descriptive interpretation are needed.
Questions for authors
-
Which central conclusions remain supported when the reported dependence-aware checks are primary and post hoc choices are treated as exploratory?
-
Under comparable correctness criteria for stated rules and bids, what evidence remains for the proposed knowledge–behavior distinction?