acceptodds
Under review as a conference paper at ICLR 2027

TradingArena: A Market-Making Benchmark for Ranking, Diagnosing, and Steering LLM Behavior

Abstract

As large language models are increasingly deployed as agents, evaluation should capture both task performance and interactive behavior. Few benchmarks jointly measure performance and behavior under controlled changes to an agent’s information, action space, and inference. We introduce **TradingArena**, a multi-agent market-making environment for both performance ranking and controlled behavioral analysis. Agents trade on a shared limit order book using noisy, biased observations of a changing hidden value, with performance measured by mechanically computed profit and loss (PnL). We evaluated 14 models across six configurations and multiple competitive rosters for performance ranking, and further used controlled interventions to diagnose behavioral failure modes. In particular, we quantitatively diagnose over-confidence in GPT-5.5, which trades aggressively under unreliable information and shows a low abstention rate on unanswerable questions. A self-refine intervention reduces this aggressiveness and moves GPT-5.5 from last to second among 14 models by mean PnL in a matched evaluation. These results illustrate how TradingArena can reveal not only differences in model outcomes, but also how behavior changes under intervention.

Primary area:
datasets and benchmarks
Keywords:
Multi-agent Environment, Benchmark, Long Horizon, State Tracking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wmXqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TradingArena is a multi-agent market-making benchmark in which LLMs trade using noisy, persistently biased observations of a changing hidden value. The paper evaluates 14 models across six market configurations and uses controlled interventions to examine their behavior. Its main case study links GPT-5.5’s aggressive trading to low abstention on unanswerable questions and shows that self-refinement reduces trading aggressiveness while improving profit.

Critical strengths
  1. The environment offers explicit control over information quality, customer behavior, and available actions. Mechanically computed profit and observable trading records make it useful for examining how changes to an agent affect both its decisions and outcomes.

  2. The paper goes beyond a leaderboard by combining interventions with an external behavioral evaluation. The differing responses to self-refinement across models are informative, and the authors appropriately acknowledge that reduced aggressiveness does not universally improve performance.

Critical weaknesses
  1. The contribution is insufficiently distinguished from closely related work. Gao et al. (2024) study LLM traders in a simulated market with order matching and controlled scenarios. Henning et al. (2025) compare homogeneous and mixed-model markets and analyze trading behavior under market interventions. Hashimoto et al. (2026) investigate behavioral biases using potentially misleading private signals and controlled information conditions. These directly relevant studies are not discussed. TradingArena’s particular combination may be useful, but the paper does not sufficiently establish what new diagnostic capability or insight it provides beyond these predecessors.

  2. The central empirical contribution remains too narrowly validated. Self-refinement adds another model call and explicitly prompts the agent to check for overreaction to recent inputs. Its benefits therefore do not clearly distinguish improved confidence calibration from additional computation or targeted prompting. The external abstention correlation provides supporting evidence but does not resolve this ambiguity. Moreover, the OpenAI evaluation stops at GPT-5.5 despite GPT-5.6 and GPT-6 Astra becoming available on July 9 and September 3, respectively, before the September 25 submission deadline (official API changelog). This does not invalidate the GPT-5.5 findings, but it weakens the claimed coverage of frontier models and leaves the relevance of the central diagnosis to newer models untested.

References

[1] Simulating Financial Market via Large Language Model based Agents

[2] LLM Agents Do Not Replicate Human Market Traders: Evidence From Experimental Finance

[3] LLM agents reveal how human bias shapes path-dependent market dynamics

Questions for authors
  1. Relative to the studies above, which specific capability does TradingArena newly enable researchers to measure, and which experiment demonstrates that added value?

  2. What was the experimental cutoff date, and what constraints prevented including GPT-5.6 or GPT-6 Astra? Are results available showing whether the main behavioral finding persists in either model?

  3. Is there a control with comparable inference compute that omits the explicit instruction about overreaction? Such a comparison would help determine whether the improvement supports the proposed diagnosis or mainly reflects additional inference.

3 comments
AuthorsOpenReview ↗

Reply to Reviewer wmXq — Part 1/3

Thank you for your comments on the benchmark contribution, model coverage, and inference controls.

Critical Weakness 1:

The contribution is insufficiently distinguished from closely related work. Gao et al. (2024) study LLM traders in a simulated market with order matching and controlled scenarios. Henning et al. (2025) compare homogeneous and mixed-model markets and analyze trading behavior under market interventions. Hashimoto et al. (2026) investigate behavioral biases using potentially misleading private signals and controlled information conditions. These directly relevant studies are not discussed. TradingArena’s particular combination may be useful, but the paper does not sufficiently establish what new diagnostic capability or insight it provides beyond these predecessors.

Question 1 (original numbering):

Relative to the studies above, which specific capability does TradingArena newly enable researchers to measure, and which experiment demonstrates that added value?

Reply:

TradingArena evaluates how LLM decisions and performance respond to changes in information, available actions, and inference procedures within a controlled competitive market. It combines a changing hidden value, private observations with separately controlled persistent bias and fresh noise, autonomous order decisions, and independent terminal settlement. Models choose order prices, quantities, and types, interact through an order book, and carry inventory. Trading records and settlement at the hidden value connect their decisions to economic outcomes independently of market quotes.

Relationship to the cited frameworks. We will add all three studies to Related Work:

  • Gao et al. simulate traders with assigned investor profiles to study economic shocks and market outcomes. Agents choose prices and quantities; holdings are valued at transaction-derived closing prices. TradingArena instead compares models and decision procedures using controlled private information, observed actions, and independent terminal settlement. Gao et al., 2024
  • Henning et al. compare human and LLM trading in an experimental-finance paradigm, including bubbles and announced dividend/redemption changes. They already provide a fundamental-value reference and interventions. TradingArena evaluates dynamic competition under imperfect private information, controlling observation errors separately from hidden-value dynamics and varying information, action, and inference conditions. Henning et al., 2025
  • Hashimoto et al. study loss aversion and herding in micro experiments, then incorporate LLM behavior into a hybrid market simulation to reproduce anomalies. Their micro experiments include misleading private signals and model-selected prices and quantities; the subsequent FCLAgent simulation uses LLM trade directions with rule-based pricing. TradingArena keeps models in control of order decisions throughout competitive play and compares models and interventions against independent terminal settlement. Hashimoto et al., 2026, Sections 3–4

Experiments demonstrating the framework’s value.

Information and environment changes. The fixed six-champion study varies persistent bias, fresh noise, value jumps, diffusion, inventory capacity, and mean reversion separately, with 20 seeds per setting. Increasing persistent-bias scale from 0 to 0.10 changes GPT-5.5’s mean PnL from 117.4 to 66.2 and Opus 4.8’s from 118.6 to 121.9, measuring different responses to information error within the same field.

Additional action capacity. With the same roster and per-order size cap of one, increasing the inventory limit from 2 to 10 changes Gemini 3.1 Pro’s mean PnL from 139.2 to 349.6 and GLM-5.2’s from 98.5 to 46.9: the same additional capacity produces different performance responses.

Inference interventions. The original study applies self-refinement separately to nine focal models in the same 14-model rich field, with ten paired seeds per model. Examples from Table 4 are:

ModelΔ taker share (percentage points)Δ mean quote sizeΔ mean PnL (rounded)
GPT-5.5−20.5−1.19+220
GPT-5.4−32.7+0.03−3
Gemini 3.1 Pro+2.1+0.30+125

The intervention produces different changes in behavior and PnL across models. The added control comparing generic refinement with the original overreaction reminder appears in Part 2.

We will clarify this framework-level contribution in Related Work and the benchmark description, distinguishing the integrated evaluation design from ingredients already present in prior work.

AuthorsOpenReview ↗

Reply to Reviewer wmXq — Part 2/3

Critical Weakness 2, excerpt on inference and interpretation:

The central empirical contribution remains too narrowly validated. Self-refinement adds another model call and explicitly prompts the agent to check for overreaction to recent inputs. Its benefits therefore do not clearly distinguish improved confidence calibration from additional computation or targeted prompting. The external abstention correlation provides supporting evidence but does not resolve this ambiguity.

Question 3 (original numbering):

Is there a control with comparable inference compute that omits the explicit instruction about overreaction? Such a comparison would help determine whether the improvement supports the proposed diagnosis or mainly reflects additional inference.

Reply:

We have added the requested two-call control to examine the role of the explicit overreaction reminder while retaining critique and revision. We will also revise the diagnostic claim: TradingArena measures trading behavior, performance, and responses to interventions, and the manuscript should present overconfidence as an interpretation supported by these observations. The external abstention correlation is supplementary evidence for that interpretation, rather than a test of the mechanism behind the refinement gain.

We evaluated three conditions with GPT-5.5 as the focal agent: A uses a single decision call; B uses a draft followed by generic critique and revision, with the recency-overreaction clause removed; C uses the original self-refinement prompt, retaining that clause. Each condition contains ten complete 100-turn games, giving 30 games with paired seeds. B and C share the two-call structure and inference settings; generated token counts vary with the responses. Only the final response is executed.

All three conditions use the same 12-model field and environment. This field differs from the original 14-model field by excluding Qwen and GLM and replacing Sonnet 4 with Sonnet 4.6; the two experiments are reported separately.

The exact second-call instructions are:

C — original self-refinement prompt:

Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, unjustified jumps, or places where you may be over-reacting to the most recent input instead of weighing the full situation you were given.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).

B — the same prompt with the recency-overreaction clause removed:

Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, or unjustified jumps.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).
ConditionMean PnLTaker shareMean quote sizeMean absolute inventory
A: single pass−135.7165.61%2.8242.379
B: generic critique/revision−36.9875.43%1.6901.740
C: original self-refinement+34.7750.97%1.2400.993

Paired PnL differences and 95% bootstrap intervals over the ten seeds are:

Paired comparisonMean ΔPnL95% bootstrap CI
C−B: retaining the recency clause+71.75[−12.65, 163.87]
B−A: generic refinement versus single pass+98.73[−57.47, 289.00]
C−A: original refinement versus single pass+170.48[3.78, 371.99]

B reduces mean order size and inventory relative to A, while its taker share increases from 65.61% to 75.43%. C has lower mean order size, inventory, and taker share than A, together with the highest mean PnL.

We will add the control, exact prompts, and behavioral results to the self-refinement analysis. Because these experiments measure the effects of a decision procedure, we will replace “quantitatively diagnose over-confidence” in the Abstract and revise “TradingArena as an Instrument” and the Conclusion to distinguish the measured trading outcomes from the overconfidence interpretation. This preserves the observed performance benefit while aligning the diagnostic wording with what the experiments measure.

AuthorsOpenReview ↗

Reply to Reviewer wmXq — Part 3/3

Critical Weakness 2, excerpt on model coverage:

Moreover, the OpenAI evaluation stops at GPT-5.5 despite GPT-5.6 and GPT-6 Astra becoming available on July 9 and September 3, respectively, before the September 25 submission deadline (official API changelog). This does not invalidate the GPT-5.5 findings, but it weakens the claimed coverage of frontier models and leaves the relevance of the central diagnosis to newer models untested.

Question 2 (original numbering):

What was the experimental cutoff date, and what constraints prevented including GPT-5.6 or GPT-6 Astra? Are results available showing whether the main behavioral finding persists in either model?

Reply:

We have addressed the model-coverage concern by adding GPT-5.6 Sol and GPT-6 Astra in two evaluations: an expanded GPT-family competition and an updated cross-family champion competition. In the family competition, the newer models show substantially lower trading aggressiveness than GPT-5.5. We re-evaluated the two newer models together with the four original GPT-family models across five configurations × 20 seeds = 100 complete games. All agents use the baseline decision interface without self-refinement.

Competitive performance. Mean final PnL over 20 seeds is:

Modelbasetoxichivolrichlong400
GPT-6 Astra511.26445.43467.83303.032536.20
GPT-5.6 Sol277.86284.42278.69167.641760.73
GPT-5.5156.12−51.91200.02−186.081142.25
GPT-5.458.6949.0372.0780.86317.73
GPT-5.244.61−45.4276.67128.2747.85
GPT-4.127.44−33.52−15.7989.69492.93

Astra ranks first and Sol second by mean PnL in every configuration.

Behavioral differences. Both newer models have lower mean taker shares and absolute inventories than GPT-5.5 in all five configurations. Taker shares are:

Modelbasetoxichivolrichlong400
GPT-5.543.9%44.8%43.2%53.0%36.8%
GPT-5.6 Sol11.6%11.3%12.7%13.4%11.3%
GPT-6 Astra6.8%6.4%7.2%10.9%5.0%

Rich also permits larger orders and a broader action set:

ModelMean PnLTaker shareMean quote sizeMean absolute inventory
GPT-5.5−186.0853.0%2.962.25
GPT-5.6 Sol+167.6413.4%1.600.97
GPT-6 Astra+303.0310.9%1.210.87

In rich, GPT-5.5’s aggressive trading and negative mean PnL recur, whereas GPT-5.6 Sol and GPT-6 Astra trade less aggressively and achieve positive mean PnL. We will describe the finding as model-dependent rather than generalizing GPT-5.5’s behavior to these newer versions.

Cross-family competition. GPT-6 Astra competes as the GPT representative alongside Opus 4.8, Gemini 3.1 Pro, and Qwen3-235B-A22B in base and rich, with 20 seeds each, giving 40 complete games. Mean final PnL is:

Modelbaserich
GPT-6 Astra75.94162.45
Opus 4.826.4634.08
Gemini 3.1 Pro189.8012.24
Qwen3-235B-A22B−93.32−77.67

Astra has the highest mean PnL in rich, while Gemini leads in base. This updated field contains four makers and one customer and is reported separately from the original six-maker, two-customer champion field. We will add both evaluations and the corresponding behavioral measurements to the manuscript, with the rosters and customer settings stated explicitly.

Reviewer_79tJOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TradingArena is a simulated multi-agent market-making environment for benchmarking LLM agents and observing trading behaviors. The authors diagnose overconfidence behavior in GPT-5.5, and with a self-refine intervention, the PnL improves.

Critical strengths

The paper provides a controllable environment and relatively thorough experiments across 14 models, market configurations, and behavioral interventions.

Critical weaknesses
  1. The authors link aggressive trading to overconfidence, but it seems not a direct measure. Table 4 shows that reduced trading aggressiveness does not consistently improve PnL. This may weaken the claim that the benchmark can diagnose specific behavioral failures.

  2. Sensitivity to competitor composition. Tables 1-3 shows model performance varies a lot across different rosters. Since the rosters also differ in some environment parameters (Appendix A.3), it raises concerns about robustness and interpretability of the model rankings.

Questions for authors
  1. Given that the ground-truth hidden values are available in the environment, could the authors directly evaluate models' uncertainty calibration by eliciting their value estimates and confidence?
  2. Are there any baseline strategies to show how well LLMs perform compared to traditional methods? E.g., strategies based on the latest price estimate, historical average estimates, some other rule-based baselines etc.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer 79tJ — Part 1/3

Thank you for your suggestions on uncertainty measurement, rule-based baselines, and the comparability of model rankings.

Question 1:

Given that the ground-truth hidden values are available in the environment, could the authors directly evaluate models' uncertainty calibration by eliciting their value estimates and confidence?

Reply:

Thank you for this suggestion. Directly eliciting estimates and confidence would assess the calibration of reported beliefs, which is distinct from our measurements of trading behavior and performance. We will make that distinction explicit and revise the claim that these trading measurements diagnose overconfidence.

Complementary evidence from controlled observation bias. Our existing bias experiment uses the hidden ground truth to vary information reliability and measure trading performance. Each agent receives a private observation s_{j,t} = θ_t + b_j + η_{j,t}, where θ_t is the hidden value, b_j is a persistent bias fixed within a game, and η_{j,t} is fresh noise. We vary the scale of the persistent biases across agents using σ_b = 0, 0.05, and 0.10, while holding the six-model roster, hidden-value process, fresh-noise setting, and other market parameters fixed. Each setting uses 20 seeds. At σ_b = 0, fresh observation noise remains, but the persistent bias is absent. Increasing σ_b introduces larger systematic observation errors, whose effect on performance can be measured because remaining inventory is ultimately settled at the true terminal value θ_T.

We highlight GPT-5.5 and Opus 4.8 from the original six-model fixed-roster experiment. Their mean PnL is:

Modelσ_b = 0σ_b = 0.05σ_b = 0.10
GPT-5.5117.4111.666.2
Opus 4.8118.6114.8121.9

GPT-5.5's mean PnL falls as persistent bias increases, whereas Opus 4.8's remains comparatively stable. This is evidence of different trading robustness to systematic observation error within the same competitive field.

We will bring this result into the behavioral discussion, replace “quantitatively diagnose over-confidence” in the Abstract, and revise the corresponding discussion and Conclusion to present overconfidence as an interpretation of the observed trading behavior.

Question 2:

Are there any baseline strategies to show how well LLMs perform compared to traditional methods? E.g., strategies based on the latest price estimate, historical average estimates, some other rule-based baselines etc.

Reply:

Following your suggestions, we evaluated two transparent rule-based strategies. Latest signal uses the current private observation as its value estimate. EMA incorporates historical observations through an exponential moving average: v̂₁ = s₁ and v̂ₜ = 0.2sₜ + 0.8v̂ₜ₋₁. Both use the same inventory-adjusted execution policy: the quote center equals the estimate minus inventory, bid and ask lie two price units below and above that center with tick rounding, order size is one, and the inventory limit is five.

Each of GPT-5.5, Opus 4.8, and Gemini 3.1 Pro plays against each baseline in a market with two makers and one customer. We use ten test seeds and swap the maker seats for every seed: 3 models × 2 baselines × 10 seeds × 2 seat assignments = 120 complete 100-turn games. We average the two assignments within each seed before calculating LLM-minus-baseline PnL differences.

Mean PnL difference (LLM − rule opponent):

LLMLatest signalEMA
GPT-5.5+581.35−86.83
Opus 4.8+325.19−121.13
Gemini 3.1 Pro+493.97+334.36

All three LLMs achieve higher mean PnL than the latest-signal baseline. In the EMA duels, the mean LLM-minus-rule differences are negative for GPT-5.5 and Opus 4.8 and positive for Gemini 3.1 Pro.

We will include both baseline definitions, their parameter-selection procedure, and the results as a separate duel evaluation. Its base conditions use κ = 0.05 and customer informedness 0.5; the PnL differences above are between opponents within these games. The EMA smoothing coefficient (α = 0.2) was chosen with knowledge of the numerical scales of observation noise and price fluctuations, whereas the LLM prompt describes these only qualitatively.

AuthorsOpenReview ↗

Reply to Reviewer 79tJ — Part 2/3

Weakness 1 — aggressive trading and the interpretation of self-refinement:

The authors link aggressive trading to overconfidence, but it seems not a direct measure. Table 4 shows that reduced trading aggressiveness does not consistently improve PnL. This may weaken the claim that the benchmark can diagnose specific behavioral failures.

Reply:

Thank you for pointing this out. Our original wording presents the overconfidence interpretation more strongly than the evidence warrants. As a controlled benchmark, TradingArena measures trading behavior, performance, and responses to interventions. GPT-5.5’s aggressive trading, losses, and response to refinement, together with the supplementary abstention results, support overconfidence as an interpretation of its behavior rather than establish it as the cause. We will revise the manuscript to make this distinction explicit.

Table 4 records different behavioral responses to the same intervention. GPT-5.4 reduces taker share by 32.7 percentage points with a rounded PnL change of −3, while Gemini 3.1 Pro gains +125 PnL with taker share increasing by 2.1 points. We will make these differences explicit in the discussion.

Additional inference control. To examine how self-refinement and its reminder about overreacting to recent input affect trading behavior and performance, we evaluated GPT-5.5 under three conditions: single-pass inference (A); a draft followed by generic critique and revision, with the overreaction reminder removed (B); and the original self-refinement procedure, which retains that reminder (C). B and C share the two-call structure and inference settings; generated token counts vary with the responses. Only the final response is executed. Each condition uses ten complete 100-turn games with paired seeds, giving 30 games.

All three conditions share the same 12-model field and environment. Compared with the original field, Qwen and GLM are excluded and Sonnet 4 is replaced by Sonnet 4.6. The second-call instructions are:

C — original self-refinement prompt:

Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, unjustified jumps, or places where you may be over-reacting to the most recent input instead of weighing the full situation you were given.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).

B — the same prompt with the recency-overreaction clause removed:

Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, or unjustified jumps.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).

The results are:

ConditionMean PnLTaker shareMean quote sizeMean absolute inventory
A: single pass−135.7165.61%2.8242.379
B: generic critique/revision−36.9875.43%1.6901.740
C: original self-refinement+34.7750.97%1.2400.993
Paired comparisonMean ΔPnL95% bootstrap CI
C−B: retaining the recency clause+71.75[−12.65, 163.87]
B−A: generic refinement versus single pass+98.73[−57.47, 289.00]
C−A: original refinement versus single pass+170.48[3.78, 371.99]

Original refinement has the highest mean PnL. B reduces mean quote size and inventory but increases taker share relative to A; C has lower means on all three behavioral indicators.

As a supplementary external comparison, nine models answered the same 912 unanswerable-question records. GPT-5.5 had the lowest judged abstention rate, 42.3%, and the highest baseline arena taker share, 72.6%; the cross-model Spearman correlation was −0.70.

We will add the control experiment and replace the Abstract’s phrase “quantitatively diagnose over-confidence” with a description of the observed trading behavior and overconfidence as an interpretation supported by these findings. In “TradingArena as an Instrument” and the Conclusion, we will foreground the measured changes in liquidity taking, order size, and PnL. The discussion of Table 4 will also make clear that models respond differently to refinement, rather than treating reduced trading aggressiveness as a general indicator of better performance or improved confidence calibration.

AuthorsOpenReview ↗

Reply to Reviewer 79tJ — Part 3/3

Weakness 2 — competitor composition and environment settings:

Sensitivity to competitor composition. Tables 1-3 shows model performance varies a lot across different rosters. Since the rosters also differ in some environment parameters (Appendix A.3), it raises concerns about robustness and interpretability of the model rankings.

Reply:

Thank you for raising this concern. In a multi-agent market, a strategy’s payoff depends on the strategies of the other participants. A strategy that performs well against strong opponents need not be the one that earns the most from opponents with exploitable weaknesses. Consequently, rankings can change with the opponent population even after averaging over random variation. Opponent-dependent performance and potentially non-transitive relationships are established considerations in multi-agent evaluation, as discussed in Re-evaluating Evaluation (Balduzzi et al., 2018) and α-Rank: Multi-Agent Evaluation by Evolution (Omidshafiei et al., 2019).

The three rosters provide complementary competitive evaluations. The champion field compares representatives of different model families; the family-internal fields compare versions within each family; and the mixed field evaluates all 14 models together against a diverse set of competitors. These evaluations reveal both advantages that persist across fields and performance that depends on the competitive setting. Rankings describe performance within each field, rather than a universal ordering independent of opponents. This is also why the paper uses the champion field as its primary overall comparison and the mixed field as a complementary evaluation.

The main finding of Gemini 3.1 Pro’s overall lead is consistent across these settings. Its results in Tables 1–3 are:

Evaluation fieldConfigurations with highest mean PnLOverall standing
Table 1: six family champions6 of 6First by average rank
Table 2: Gemini-family competition5 of 5First in every configuration
Table 3: 14-model mixed field5 of 6First by average rank

The exception in Table 3 is rich, where GLM-5.2 has the highest mean PnL. Each configuration is evaluated over 20 simulation seeds, and rankings are based on mean final PnL. These repeated evaluations reduce the influence of individual game outcomes, while the different rosters test performance in different competitive contexts.

Tables 1–3 use the same underlying market mechanism and configuration framework, with field-specific parameters documented in Appendix A.3. Our fixed-roster sensitivity study further examines the effects of individual environment parameters over 20 seeds per setting. We will clarify how these controlled comparisons complement the evaluations across opponent populations.

Thank you for the detailed response and additional experiments. The ranking robustness concern remains partially unresolved, as competitor rosters also differ in environment parameters. Overall, the rebuttal addresses most of my concerns, and I raise my rating.

Reviewer_Hr2pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TradingArena evaluates LLM agents through interactive trading and controlled market conditions. The evaluation across multiple models and configurations is informative, and the analysis of behavioural changes under intervention is a useful contribution. The authors evaluate 14 models across six market configurations and investigate differences in trading behaviour. A case study examines whether aggressive trading reflects model overconfidence and whether a self-refinement procedure can improve performance. However, the paper does not convincingly establish that the observed trading behaviour reflects general capabilities such as overconfidence or long-horizon reasoning.

Critical strengths

TradingArena provides a useful framework for evaluating LLM agents through observable decisions and measurable financial outcomes. The paper evaluates multiple model families, market configurations, and competitive settings. The behavioural analysis goes beyond model rankings by examining trading aggressiveness, responses to self-refinement, and performance changes under controlled interventions.

Critical weaknesses

The paper treats aggressive trading as a sign of overconfidence, but it could also reflect different trading strategies or market conditions. Although self-refinement improves performance, this does not prove that overconfidence was the problem. Stronger evidence is needed to support this interpretation. TradingArena relies entirely on a simplified simulated market, without demonstrating whether its model rankings and behavioural findings generalise to real financial markets. The paper does not discuss Ma et al. (Findings of EMNLP 2025), which already introduced a multi-agent trading benchmark examining numerical reasoning, reflection, and trading performance. The authors should clarify what TradingArena adds beyond this work. Furthermore, the evaluation relies entirely on a simplified simulated market, without showing whether its findings generalise to realistic trading conditions.

Questions for authors

How can the authors distinguish overconfidence from other explanations for aggressive trading, such as different risk preferences or trading strategies?

Does self-refinement improve performance because it reduces overconfidence, or because it provides additional inference and changes the agent's trading strategy?

4 comments
AuthorsOpenReview ↗

Reply to Reviewer Hr2p — Part 1/4

Thank you for your comments on the behavioral interpretation and evaluation design.

Related Critical Weakness (addressed in Questions 1 and 2):

The paper treats aggressive trading as a sign of overconfidence, but it could also reflect different trading strategies or market conditions. Although self-refinement improves performance, this does not prove that overconfidence was the problem. Stronger evidence is needed to support this interpretation.

Question 1:

How can the authors distinguish overconfidence from other explanations for aggressive trading, such as different risk preferences or trading strategies?

Reply:

We interpret GPT-5.5’s aggressive trading, losses, and response to refinement as evidence consistent with overconfidence. Risk preferences and trading strategies are also possible explanations, and we will make this distinction explicit in the manuscript.

Trading behavior and intervention responses. In the original 14-model rich field, GPT-5.5 has a baseline taker share of 72.6% and mean quote size of 2.45. Self-refinement changes its mean PnL from −164.93 to +54.58, moving it from last to second by mean PnL in the corresponding fields. Taker share falls by 20.5 percentage points, and mean quote size falls to 1.26. These comparisons use ten paired seeds and the same roster and environment settings. The added instruction asks the agent to critique and revise its decision, including possible overreaction to recent input. We also conducted 30 additional games comparing single-pass inference, generic self-refinement, and the original self-refinement procedure with its overreaction reminder; the results and exact prompts are provided in Question 2.

Supplementary external measurement. In the original study, nine models also answered the same 912 unanswerable-question records from AbstentionBench. Among non-empty responses with valid Opus 4.8 judgments, GPT-5.5 has the lowest abstention rate, 42.3%. Across these models, abstention and baseline arena taker share have a Spearman correlation of −0.70.

Manuscript revisions. As a controlled benchmark, TradingArena measures trading behavior and the effects of interventions; these observations inform behavioral explanations without uniquely identifying their cause. Our wording should reflect that distinction. In the Abstract, we will replace “quantitatively diagnose over-confidence” with a description of GPT-5.5’s aggressive trading and overconfidence as a possible explanation. In “TradingArena as an Instrument” and the Conclusion, we will lead with the changes in liquidity taking, order size, and PnL, with external abstention as supplementary evidence. We will also revise the rich-configuration description to focus on how agents use additional action flexibility and how their aggressiveness relates to PnL.

Regarding long-horizon reasoning, our Planning appendix evaluates an instruction that removes explicit horizon cues and requests current-turn optimization, using 20 games per model. We will describe the resulting performance differences as model-dependent responses to horizon information and trading objectives, consistent with the scope of this intervention.

AuthorsOpenReview ↗

Reply to Reviewer Hr2p — Part 2/4

Question 2:

Does self-refinement improve performance because it reduces overconfidence, or because it provides additional inference and changes the agent's trading strategy?

Reply:

To examine the role of the overreaction reminder, we added a control that gives GPT-5.5 a second call to critique and revise its decision while removing that reminder from the original prompt. Across the three conditions, the original self-refinement procedure achieves the highest mean PnL. Compared with single-pass inference, it improves PnL while reducing liquidity taking, order size, and inventory exposure. We will report these findings as measured effects of self-refinement on trading behavior and performance.

We label these three conditions A: single-pass inference, B: generic two-call refinement, and C: original two-call refinement. C−A measures the effect of the original procedure as a whole; B−A measures the effect of generic refinement; C−B measures the incremental effect of retaining the reminder under the same two-call structure and inference settings.

Each condition contains ten complete 100-turn games with GPT-5.5 as the focal agent, giving 30 games with paired seeds. B and C share the two-call structure and inference settings; generated token counts vary with the responses. Only the final response is executed.

All three conditions use the same 12-model field and environment. This field differs from the original 14-model field by excluding Qwen and GLM and replacing Sonnet 4 with Sonnet 4.6; the two experiments are reported separately.

C — original self-refinement prompt:

Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, unjustified jumps, or places where you may be over-reacting to the most recent input instead of weighing the full situation you were given.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).

B — the same prompt with the recency-overreaction clause removed:

Before this is submitted, act as a critic of your own proposed response above.
1. CRITIQUE it: point out any concrete mistakes, internal inconsistencies, or unjustified jumps.
2. Then produce an IMPROVED final response that addresses your critique. If the original was already best, you may keep it.
Respond with ONLY a raw JSON object using the SAME schema as before (thoughts, optional scratchpad/scratchpad_append, commands).
ConditionMean PnLTaker shareMean quote sizeMean absolute inventory
A: single pass−135.7165.61%2.8242.379
B: generic critique/revision−36.9875.43%1.6901.740
C: original self-refinement+34.7750.97%1.2400.993

Paired PnL differences and 95% bootstrap intervals over the ten seeds are:

Paired comparisonMean ΔPnL95% bootstrap CI
C−B: retaining the recency clause+71.75[−12.65, 163.87]
B−A: generic refinement versus single pass+98.73[−57.47, 289.00]
C−A: original refinement versus single pass+170.48[3.78, 371.99]

The behavioral responses differ across conditions. B reduces mean order size and inventory relative to A, while taker share increases from 65.61% to 75.43%. C has lower mean order size, inventory, and taker share than A, together with the highest mean PnL.

We will add the control and exact prompts to the self-refinement analysis and describe the measured changes in behavior and performance. The Abstract and Conclusion will present overconfidence as an interpretation of the behavioral pattern, rather than an established mechanism of the performance gain.

AuthorsOpenReview ↗

Reply to Reviewer Hr2p — Part 3/4

Weakness 1 — relationship to Ma et al.:

The paper does not discuss Ma et al. (Findings of EMNLP 2025), which already introduced a multi-agent trading benchmark examining numerical reasoning, reflection, and trading performance. The authors should clarify what TradingArena adds beyond this work.

Reply:

Thank you for pointing us to Ma et al.'s Agent Trading Arena: A Study on Numerical Understanding in LLM-Based Agents. It develops a virtual market in which agent interactions drive price dynamics, and studies numerical understanding under textual versus visual inputs and strategy improvement through reflection. Ma et al., 2025

The central distinction is the separation of settlement value, private information, and trading prices in our evaluation protocol. TradingArena generates a dynamic hidden value, θ_t, independently of agents' orders. Agent j receives a private observation s_{j,t} = θ_t + b_j + η_{j,t}, where b_j is a persistent bias fixed within a game and η_{j,t} is fresh observation noise. Agents choose their order prices and quantities using this imperfect information and the observed market state. Transactions occur through limit-order-book matching, and terminal inventory is valued at the hidden settlement value θ_T.

This separation gives the evaluation an independent settlement reference: terminal PnL equals the change in cash plus remaining inventory multiplied by θ_T. An increase in quoted or transaction prices does not by itself raise the terminal valuation of remaining inventory. Agents' use of unreliable information therefore has measurable economic consequences through their execution prices and positions.

We can therefore vary persistent observation bias, fresh noise, and customer informedness separately, and measure the resulting changes in trading behavior and PnL while holding the roster and other settings fixed.

For example, the original fixed-roster parameter study varies six environment parameters one at a time, with 20 seeds per setting. Holding the original six-champion roster and other settings fixed, varying the scale of persistent observation bias gives the following mean PnL:

Modelσ_b = 0σ_b = 0.05σ_b = 0.10
GPT-5.5117.4111.666.2
Opus 4.8118.6114.8121.9
Gemini 3.1 Pro234.4212.9242.3
Sonnet 475.427.957.0
Qwen3-235B−5.6−69.8−86.1
GLM-5.2146.078.5101.6

As the bias scale increases, GPT-5.5's mean PnL falls from 117.4 to 66.2, whereas Opus 4.8's means range from 114.8 to 121.9. This experiment illustrates the added evaluation capability: measuring model-specific performance responses to a controlled source of information error within the same competitive field.

We will add Ma et al. to Related Work and explain how our independent settlement value and controlled information errors enable the comparisons above.

AuthorsOpenReview ↗

Reply to Reviewer Hr2p — Part 4/4

Weakness 2 — real-market generalization:

TradingArena relies entirely on a simplified simulated market, without demonstrating whether its model rankings and behavioural findings generalise to real financial markets.

Reply:

The simplified market is an intentional design choice: it lets us measure how models use imperfect information and respond to controlled changes in the environment. The resulting rankings describe performance in these settings; transfer to real markets requires separate evaluation.

Why the evaluation target matters. Asking an agent to trade directly from news and prices evaluates a combination of information acquisition, signal interpretation, position sizing, and execution under a particular market regime. Returns alone do not identify which component accounts for a performance difference, and market exposure and random fluctuations can obscure differences in decision quality. Alternatively, asking an agent to build a quantitative trading system from scratch also evaluates data engineering, coding, strategy research, and backtesting. Both are valuable tasks, but they introduce additional capabilities and system choices beyond the decision behaviors we seek to examine.

Recent studies motivate this distinction. Li et al.'s Can LLM-based Financial Investing Strategies Outperform the Market in Long Run? (FINSABER, KDD 2026) finds that previously reported advantages of LLM investing strategies weaken under longer and broader evaluations. In its Random Five setting, buy-and-hold achieves a Sharpe ratio of 0.315, compared with −0.253 for FinMem and 0.094 for FinAgent; results also depend on the stock-selection setting and market regime. Li et al., 2026

Fan et al.'s AI-Trader: Benchmarking Autonomous Agents in Real-Time Financial Markets likewise reports substantial cross-market variation: in its evaluation windows, three of six models outperform the U.S. equity baseline in cumulative return, none outperform the A-share baseline, and one outperforms the cryptocurrency baseline. Fan et al., 2025

TradingArena gives models a common decision interface and private observations with controlled sources of error. They face partial observability, order-book competition, and inventory risk, but do not need to build a data pipeline or trading system. We can vary information reliability or action constraints while holding other settings fixed, and evaluate their decisions against an independent terminal settlement value.

The fixed-roster bias experiment described in Part 3 illustrates this control: it isolates a specified change in information reliability that would be difficult to separate from other influences in a real-market comparison.

We will explain this rationale in the Introduction and environment description, add the above studies to Related Work, and distinguish controlled behavioral evaluation from evaluation of complete trading systems on real-market data.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.