- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper conducts algorithmic pricing experiments in which two agents set wages in a logit supply model. First, they conduct experiments with Q-learning agents and observe wage suppression, similarly to the findings of Calvano et al.. (2020). Then they conduct experiments with 8 open weight LLMs. The LLMs exhibit heterogeneous wage-setting behavior. The main results involve the three Qwen 3 models, which have a reasoning toggle. They find that enabling reasoning results in wage suppression, across all three Qwen 3 sizes.
Critical strengths
- I think the main question/concern, about the extent to which reasoning influences pricing behavior, is interesting. The authors make a valid point that LLM safety properties are typically only evaluated at a single reasoning setting. But for settings like pricing, one might expect different reasoning settings to nontrivially influence pricing behavior (as they demonstrate).
Critical weaknesses
-
Oligopsony wage-setting requires relatively advanced capabilities: for example, sufficient quantitative reasoning skills to understand the past wage data and make inferences about the demand environment, and sufficient strategic reasoning skills to be able to interpret the competitor's actions and respond accordingly. I am concerned that the experiment design does not elicit sufficient capabilities from the LLMs so that their behavior can be meaningfully interpreted. First regarding the harness, the LLM agents here only get access to previous wage data. They do not have any freetext memory that persists between periods, even though prior LLM studies such as Fish et al. (2026) have shown that a text-based memory is important to elicit sufficient capabilities. Second regarding the choice of LLMs themselves, it is unclear whether the Qwen 3 models are sufficiently capable. Here too it would be better to repeat the monopoly test from Fish et al. (2026) to ensure the analyzed models are sufficiently capable.
-
I wish the analysis in Appendix G was expanded more. The finding that reasoning suppresses wages in Qwen 3 models seems pretty robust (though I would have liked to see this replicated on other model families -- this limitation the authors already mention in the paper). But I am not really left with any understanding of why this happens. Because of the finding that in the dual Calvano/Fish pricing game, reasoning causes prices to fall, I am left wondering whether the effect is just a spurious preference for lower numbers, that will not replicate on more capable LLMs.
Questions for authors
What are the results of running the monopoly experiment from Fish et al. (2026) on these models?
3 comments
Thank you for running the Fish-style monopoly experiment. Unfortunately, my interpretation is that the results do not bear good news for your paper. You report that Qwen 32B with reasoning ON reaches >99% of optimal profit, which is a pass. But Qwen 32B with reasoning OFF, as well as 8B under both ON/OFF, clearly fail this simple bar. (I don't see results for 14B in the appendix, so maybe you didn't have the resources to run them, which I am fine with. My guess is that they would fail.)
So then the trouble is, your main experiments are studying the wage-setting behavior of Qwen 8B, 14B, and 32B models, with and without reasoning. But this Fish-style adaptation shows that only Qwen 32B with reasoning ON is sufficiently capable to solve the very simple problem of finding the optimal wage in a static environment. All of the other models are not. So I am not really sure how one can interpret the wage-setting behavior of the other models, if they are not able to complete this simple task.
To make the comparisons in this paper meaningful, the results should analyze the data of LLMs that meet this minimum capability bar. Unfortunately I understand that this requires substantial revisions beyond the scope of a typical revision phase, which is why I maintain my recommendation to reject.
Thank you again for clarifying the capability concern (and for your genuine encouragement, which really helps). We conducted an additional evaluation directly in the paper’s labor-market environment, including more capable Qwen3.8-27B.
We fixed the other firm’s wage at three pre-specified levels and ran 10 independent seeds for 100 rounds per configuration, totaling 12,000 wage decisions. The table reports mean profit over the pre-specified final 30 rounds as a percentage of exact best-response profit. The optimal wage was not supplied to the agents.
| Model | Reasoning | Rival wage 1.075 | Rival wage 1.527 | Rival wage 1.900 |
|---|---|---|---|---|
| Qwen3-32B | OFF | 81.99% | 97.85% | 36.70% |
| Qwen3-32B | ON | 99.06% | 99.76% | 88.00% |
| Qwen3.8-27B | OFF | 89.03% | 81.28% | 85.13% |
| Qwen3.8-27B | ON | 99.56% | 98.58% | 99.54% |
Qwen3.8-27B ON passes in two of three conditions under a conservative criterion (a simultaneous 95% confidence lower bound of at least 99% of optimal profit.) Its remaining condition averages 98.58% and is statistically inconclusive relative to that threshold. Qwen3-32B ON also passes at the middle rival wage.
These results establish reliably near-optimal performance in multiple configurations of the original wage environment. They also identify limitations: no model-reasoning configuration passes across all three conditions, and these checks alone do not validate strategic interpretations of the multi-agent results.
The findings concern the tested interface, including a 10-round observation history and a 1,024-token reasoning cap. Cross-model comparisons are descriptive, as Qwen3-32B used bfloat16 and Qwen3.8-27B used float32.
We will incorporate these diagnostics and qualify interpretations where capability remains unestablished. Understanding how reasoning and optimization limitations shape wages still remains a consequential research question. Explicit capability screening strengthens the empirical basis for studying that question and interpreting its implications for each affected parties.
Thank you for the careful request on capability elicitation and persistent memory. We added a Fish-style monopoly test and a memory experiment in the wage game. In the revised PDF, purple marks changes addressing your review; red marks changes shared across multiple reviews. Footnotes identify reviewer IDs.
1. Results of the Fish-style monopoly experiment. Reasoning-enabled 32B reaches 99.16% of optimal profit with carried memory and 99.25% without it, averaged over the final 100 periods. The corresponding OFF values are 64.69% and 65.94%. The 8B model performs substantially worse. This gives us direct evidence of 32B's quantitative capability on this task, although your concern about 8B understandably remains.
Qwen3-8B/32B each run for 300 periods with reasoning ON and OFF, memory carry and erase, and through three currency scales. Agents receive price, quantity, and profit history; the carry condition also retains plans and insights. Demand is unknown to the agent. The reported means weight the scales equally, with one seed per scale, so they are descriptive averages rather than independent-seed estimates with confidence intervals.
Added Appendix J.3 documents the prompt, decoding, and sampling differences from Fish; Table 19 reports every condition. We call this a Fish-style adaptation (since it is not an exact replication.) Success here does not by itself establish multi-agent strategic competence, but it answers the capability question more directly than wage trajectories alone.
2. Persistent memory in the original wage game. We added a bounded recursive private note alongside the numerical history. Both carry and erase conditions generate notes; only carry condition passes the note to the next decision. This controls for note generation while changing access to persistent text.
Across ten paired seeds per model, with equal model weights, the carry-memory ON-minus-OFF wage effect is -0.053, with 95% CI [-0.162, +0.057]. The average remains negative, though the interval includes zero. Employer surplus rises by 32.62%, with an absolute-effect CI [+0.043, +0.208] and Holm-adjusted p = 0.034. In other words, the wage estimate becomes less conclusive under this harness, while the employer-surplus effect remains statistically supported. Appendix J.4 gives the matched contrasts and all welfare outcomes.
3. A preference for lower numbers We reversed action coordinates so that lower actual prices require explicitly larger outputs, while preserving economic consequences. Reasoning still lowers prices by -0.406, with significant CI [-0.515, -0.297]. A uniform smaller-output preference cannot explain this result. Reversed-coordinate wages instead rise, so wage suppression is sensitive to representation (Appendix J.5).
Appendix G expands the trace analysis, including profit arguments and quantitative errors, without claiming a unique internal mechanism. We separate the original stored-action findings from follow-ups with explicit final-answer extraction. Missing original raw generations prevent retrospective extraction checks. Other-family expansion remains outside this revision.
Thank you again for pushing us to test the harness more carefully. The core catalyst stands the same, in that workers can unintentionally bear the consequences of an employer's LLM delegation without ever interacting with the system itself.