- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper audits whether language models acting as purchasing agents hold coherent preferences, using the weakest requirement of economic rationality: a monotone decision boundary in price. The method sweeps dense price grids while holding the transaction fixed (about 137M prompts, 1.1B single-token greedy yes/no decisions), classifies each decision curve (constant, near-constant, or containing a step), fits a logistic boundary with instability metrics, and tests invariance under paraphrase, quantity changes, and economically irrelevant buyer/seller names and occupations. Eight open-weight models (0.5B to 32B) are scored on a coherence leaderboard. Findings: coherent step curves are the exception (30 of 64 baseline curves); equivalent paraphrases move boundaries by factors above 5; replacing generic roles with names shifts demand by up to 95 percentage points (Llama-3.1-8B on water: 24 percent generic versus 94 percent for the median named pair) with boundary ratios up to 16.5x; quantity direction is always correct; half the models score worse than an agent that never changes its decision.
Critical strengths
The empirical execution is thorough in ways that matter: name-level permutation tests, a replication with a second validated name set, threshold sensitivity analyses, and a temperature ablation are all present, and the headline name effect is large enough (24 to 94 percent) to be robust to reasonable analytic choices. The documentation of identity-sensitive purchasing decisions is a finding of real concern and, to my knowledge, new at this granularity.
Critical weaknesses
-
The gap between what is measured and what is claimed is large, and it is consequential because the paper's framing ("economic alignment", a coherence leaderboard) invites conclusions the data cannot support. The task is a single-shot yes/no purchase of water or wheat with no budget, no alternatives, and no interaction, and the model pool contains only open-weight models up to 32B. Deployed purchasing agents are built on frontier closed models, none of which are tested, although the method needs only a yes/no API call. As a result the leaderboard, the most visible artifact of the paper, ranks models nobody deploys for this purpose on a task far below deployment complexity.
-
The main results use greedy decoding, and the paper's own Appendix I shows this choice is load-bearing: at temperature 1, boundaries disappear for five of the seven panels that had one, and turbulence rises to 0.2 to 0.5 for six of eight models. Since deployment uses sampling, the main tables likely overstate coherence, and the ablation that shows this should be in the main text rather than the appendix. The authors are candid that the leaderboard is "a summary, not a test" with no uncertainty estimate, which further limits what the ranking can support. I also note Figure 1 is not legible at print size (16 compressed panels whose axis labels and annotations cannot be read), which matters because it is the main qualitative evidence for the curve taxonomy.
Questions for authors
-
Can you run the price sweep on at least one frontier closed model? The per-decision cost is a single output token, and this would determine whether the incoherence findings describe deployed systems or only small open-weight models.
-
Would the leaderboard ranking survive at the sampling temperatures used in deployment (e.g. T = 0.7)? If not, which results should readers treat as the operative ones?
-
For the panels that are saturated (always buy or never buy), how much of the name-effect analysis do they carry?
Thank you for your time and efforts for a detailed review. Your recognition of the thoroughness of the empirical work, including the permutation tests, replication and ablations, and of the identity-sensitive purchasing findings, which you described as new at this granularity, is greatly appreciated. Also, your points on the gap between the task and deployed agents, the role of decoding temperature, and the clarity of the presentation are valuable. This submission will be withdrawn so that we can address your points properly in our next revision. Thank you again for your thoughtful contribution.