- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TradingArena is a multi-agent market-making benchmark in which LLMs trade using noisy, persistently biased observations of a changing hidden value. The paper evaluates 14 models across six market configurations and uses controlled interventions to examine their behavior. Its main case study links GPT-5.5’s aggressive trading to low abstention on unanswerable questions and shows that self-refinement reduces trading aggressiveness while improving profit.
Critical strengths
-
The environment offers explicit control over information quality, customer behavior, and available actions. Mechanically computed profit and observable trading records make it useful for examining how changes to an agent affect both its decisions and outcomes.
-
The paper goes beyond a leaderboard by combining interventions with an external behavioral evaluation. The differing responses to self-refinement across models are informative, and the authors appropriately acknowledge that reduced aggressiveness does not universally improve performance.
Critical weaknesses
-
The contribution is insufficiently distinguished from closely related work. Gao et al. (2024) study LLM traders in a simulated market with order matching and controlled scenarios. Henning et al. (2025) compare homogeneous and mixed-model markets and analyze trading behavior under market interventions. Hashimoto et al. (2026) investigate behavioral biases using potentially misleading private signals and controlled information conditions. These directly relevant studies are not discussed. TradingArena’s particular combination may be useful, but the paper does not sufficiently establish what new diagnostic capability or insight it provides beyond these predecessors.
-
The central empirical contribution remains too narrowly validated. Self-refinement adds another model call and explicitly prompts the agent to check for overreaction to recent inputs. Its benefits therefore do not clearly distinguish improved confidence calibration from additional computation or targeted prompting. The external abstention correlation provides supporting evidence but does not resolve this ambiguity. Moreover, the OpenAI evaluation stops at GPT-5.5 despite GPT-5.6 and GPT-6 Astra becoming available on July 9 and September 3, respectively, before the September 25 submission deadline (official API changelog). This does not invalidate the GPT-5.5 findings, but it weakens the claimed coverage of frontier models and leaves the relevance of the central diagnosis to newer models untested.
References
[1] Simulating Financial Market via Large Language Model based Agents
[2] LLM Agents Do Not Replicate Human Market Traders: Evidence From Experimental Finance
[3] LLM agents reveal how human bias shapes path-dependent market dynamics
Questions for authors
-
Relative to the studies above, which specific capability does TradingArena newly enable researchers to measure, and which experiment demonstrates that added value?
-
What was the experimental cutoff date, and what constraints prevented including GPT-5.6 or GPT-6 Astra? Are results available showing whether the main behavioral finding persists in either model?
-
Is there a control with comparable inference compute that omits the explicit instruction about overreaction? Such a comparison would help determine whether the improvement supports the proposed diagnosis or mainly reflects additional inference.
3 comments
Reply to Reviewer wmXq — Part 2/3
Critical Weakness 2, excerpt on inference and interpretation:
The central empirical contribution remains too narrowly validated. Self-refinement adds another model call and explicitly prompts the agent to check for overreaction to recent inputs. Its benefits therefore do not clearly distinguish improved confidence calibration from additional computation or targeted prompting. The external abstention correlation provides supporting evidence but does not resolve this ambiguity.
Question 3 (original numbering):
Is there a control with comparable inference compute that omits the explicit instruction about overreaction? Such a comparison would help determine whether the improvement supports the proposed diagnosis or mainly reflects additional inference.
Reply:
We have added the requested two-call control to examine the role of the explicit overreaction reminder while retaining critique and revision. We will also revise the diagnostic claim: TradingArena measures trading behavior, performance, and responses to interventions, and the manuscript should present overconfidence as an interpretation supported by these observations. The external abstention correlation is supplementary evidence for that interpretation, rather than a test of the mechanism behind the refinement gain.
We evaluated three conditions with GPT-5.5 as the focal agent: A uses a single decision call; B uses a draft followed by generic critique and revision, with the recency-overreaction clause removed; C uses the original self-refinement prompt, retaining that clause. Each condition contains ten complete 100-turn games, giving 30 games with paired seeds. B and C share the two-call structure and inference settings; generated token counts vary with the responses. Only the final response is executed.
All three conditions use the same 12-model field and environment. This field differs from the original 14-model field by excluding Qwen and GLM and replacing Sonnet 4 with Sonnet 4.6; the two experiments are reported separately.
The exact second-call instructions are:
C — original self-refinement prompt:
Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, unjustified jumps, or places where you may be over-reacting to the most recent input instead of weighing the full situation you were given.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).
B — the same prompt with the recency-overreaction clause removed:
Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, or unjustified jumps.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).
| Condition | Mean PnL | Taker share | Mean quote size | Mean absolute inventory |
|---|---|---|---|---|
| A: single pass | −135.71 | 65.61% | 2.824 | 2.379 |
| B: generic critique/revision | −36.98 | 75.43% | 1.690 | 1.740 |
| C: original self-refinement | +34.77 | 50.97% | 1.240 | 0.993 |
Paired PnL differences and 95% bootstrap intervals over the ten seeds are:
| Paired comparison | Mean ΔPnL | 95% bootstrap CI |
|---|---|---|
| C−B: retaining the recency clause | +71.75 | [−12.65, 163.87] |
| B−A: generic refinement versus single pass | +98.73 | [−57.47, 289.00] |
| C−A: original refinement versus single pass | +170.48 | [3.78, 371.99] |
B reduces mean order size and inventory relative to A, while its taker share increases from 65.61% to 75.43%. C has lower mean order size, inventory, and taker share than A, together with the highest mean PnL.
We will add the control, exact prompts, and behavioral results to the self-refinement analysis. Because these experiments measure the effects of a decision procedure, we will replace “quantitatively diagnose over-confidence” in the Abstract and revise “TradingArena as an Instrument” and the Conclusion to distinguish the measured trading outcomes from the overconfidence interpretation. This preserves the observed performance benefit while aligning the diagnostic wording with what the experiments measure.
Reply to Reviewer wmXq — Part 3/3
Critical Weakness 2, excerpt on model coverage:
Moreover, the OpenAI evaluation stops at GPT-5.5 despite GPT-5.6 and GPT-6 Astra becoming available on July 9 and September 3, respectively, before the September 25 submission deadline (official API changelog). This does not invalidate the GPT-5.5 findings, but it weakens the claimed coverage of frontier models and leaves the relevance of the central diagnosis to newer models untested.
Question 2 (original numbering):
What was the experimental cutoff date, and what constraints prevented including GPT-5.6 or GPT-6 Astra? Are results available showing whether the main behavioral finding persists in either model?
Reply:
We have addressed the model-coverage concern by adding GPT-5.6 Sol and GPT-6 Astra in two evaluations: an expanded GPT-family competition and an updated cross-family champion competition. In the family competition, the newer models show substantially lower trading aggressiveness than GPT-5.5. We re-evaluated the two newer models together with the four original GPT-family models across five configurations × 20 seeds = 100 complete games. All agents use the baseline decision interface without self-refinement.
Competitive performance. Mean final PnL over 20 seeds is:
| Model | base | toxic | hivol | rich | long400 |
|---|---|---|---|---|---|
| GPT-6 Astra | 511.26 | 445.43 | 467.83 | 303.03 | 2536.20 |
| GPT-5.6 Sol | 277.86 | 284.42 | 278.69 | 167.64 | 1760.73 |
| GPT-5.5 | 156.12 | −51.91 | 200.02 | −186.08 | 1142.25 |
| GPT-5.4 | 58.69 | 49.03 | 72.07 | 80.86 | 317.73 |
| GPT-5.2 | 44.61 | −45.42 | 76.67 | 128.27 | 47.85 |
| GPT-4.1 | 27.44 | −33.52 | −15.79 | 89.69 | 492.93 |
Astra ranks first and Sol second by mean PnL in every configuration.
Behavioral differences. Both newer models have lower mean taker shares and absolute inventories than GPT-5.5 in all five configurations. Taker shares are:
| Model | base | toxic | hivol | rich | long400 |
|---|---|---|---|---|---|
| GPT-5.5 | 43.9% | 44.8% | 43.2% | 53.0% | 36.8% |
| GPT-5.6 Sol | 11.6% | 11.3% | 12.7% | 13.4% | 11.3% |
| GPT-6 Astra | 6.8% | 6.4% | 7.2% | 10.9% | 5.0% |
Rich also permits larger orders and a broader action set:
| Model | Mean PnL | Taker share | Mean quote size | Mean absolute inventory |
|---|---|---|---|---|
| GPT-5.5 | −186.08 | 53.0% | 2.96 | 2.25 |
| GPT-5.6 Sol | +167.64 | 13.4% | 1.60 | 0.97 |
| GPT-6 Astra | +303.03 | 10.9% | 1.21 | 0.87 |
In rich, GPT-5.5’s aggressive trading and negative mean PnL recur, whereas GPT-5.6 Sol and GPT-6 Astra trade less aggressively and achieve positive mean PnL. We will describe the finding as model-dependent rather than generalizing GPT-5.5’s behavior to these newer versions.
Cross-family competition. GPT-6 Astra competes as the GPT representative alongside Opus 4.8, Gemini 3.1 Pro, and Qwen3-235B-A22B in base and rich, with 20 seeds each, giving 40 complete games. Mean final PnL is:
| Model | base | rich |
|---|---|---|
| GPT-6 Astra | 75.94 | 162.45 |
| Opus 4.8 | 26.46 | 34.08 |
| Gemini 3.1 Pro | 189.80 | 12.24 |
| Qwen3-235B-A22B | −93.32 | −77.67 |
Astra has the highest mean PnL in rich, while Gemini leads in base. This updated field contains four makers and one customer and is reported separately from the original six-maker, two-customer champion field. We will add both evaluations and the corresponding behavioral measurements to the manuscript, with the rosters and customer settings stated explicitly.
Reply to Reviewer wmXq — Part 1/3
Thank you for your comments on the benchmark contribution, model coverage, and inference controls.
Critical Weakness 1:
Question 1 (original numbering):
Reply:
TradingArena evaluates how LLM decisions and performance respond to changes in information, available actions, and inference procedures within a controlled competitive market. It combines a changing hidden value, private observations with separately controlled persistent bias and fresh noise, autonomous order decisions, and independent terminal settlement. Models choose order prices, quantities, and types, interact through an order book, and carry inventory. Trading records and settlement at the hidden value connect their decisions to economic outcomes independently of market quotes.
Relationship to the cited frameworks. We will add all three studies to Related Work:
Experiments demonstrating the framework’s value.
Information and environment changes. The fixed six-champion study varies persistent bias, fresh noise, value jumps, diffusion, inventory capacity, and mean reversion separately, with 20 seeds per setting. Increasing persistent-bias scale from 0 to 0.10 changes GPT-5.5’s mean PnL from 117.4 to 66.2 and Opus 4.8’s from 118.6 to 121.9, measuring different responses to information error within the same field.
Additional action capacity. With the same roster and per-order size cap of one, increasing the inventory limit from 2 to 10 changes Gemini 3.1 Pro’s mean PnL from 139.2 to 349.6 and GLM-5.2’s from 98.5 to 46.9: the same additional capacity produces different performance responses.
Inference interventions. The original study applies self-refinement separately to nine focal models in the same 14-model rich field, with ten paired seeds per model. Examples from Table 4 are:
The intervention produces different changes in behavior and PnL across models. The added control comparing generic refinement with the original overreaction reminder appears in Part 2.
We will clarify this framework-level contribution in Related Work and the benchmark description, distinguishing the integrated evaluation design from ingredients already present in prior work.