acceptodds
Desk rejected at ICLR 2027

The Interface Is the Policy: Output Typing, Calibration, and Confidence-Gated Routing in LLM Trading Agents

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM) trading agents are evaluated as if the model were the policy: frameworks differ in memory, reflection, and multi-agent structure, but nearly all terminate in a generated decision that a parser reduces to buy, sell, or hold. We argue that this final step, the output interface, is a design axis whose effect on decision quality has never been isolated. We formalise an interface ladder of four rungs that hold the model, prompt, and market context fixed while varying only what the model is asked to emit: regex-parsed free text (L0), a constrained enum (L1), an enum with verbalized confidence (L2), and a typed probability vector over next-day direction (L3). The ladder exposes a structural fact that prior benchmarks obscure: rungs L0–L2 can only be mapped to a full-size long, short, or flat position, whereas L3 admits fractional sizing, so any advantage of “primitivized” outputs must flow through the probability-to-position map rather than the label. We instantiate the ladder on InvestorBench, an open benchmark of seven equities and cryptocurrencies, with open-weight backbones from 7B to 72B; audit whether the confidence emitted by L2 and L3 is actionable, meaning calibrated, discriminative, and stable across assets; and sweep the two-threshold “auto-execute above τ_hi, escalate below τ_lo” routing rule against always-execute. Asking the same model for a different output type changes its direction on roughly half of all days (agreement with the enum rung 0.52, κ = 0.04), so the interface is part of the policy, yet no rung has detectably better risk-adjusted return under sign-only sizing. Mapping L3’s probability vector to a fractional position cuts volatility to 0.25× and drawdown to 0.21× at equal or better Sharpe, which is where typed outputs earn their keep. The emitted confidence is honest after temperature scaling (ECE 0.117 → 0.047) but not discriminative (AUROC = 0.530), so no confidence-gated routing policy dominates always-execute. Code, prompts, decision logs, and the harness are released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, trading, output interface, confidence calibration, routing

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 4 reviews, above 0% of the others.
Reviewer_WC2nOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an output-interface ladder for trading agents, from free text to typed probability vectors, and studies position sizing, confidence calibration, and escalation routing. These are worthwhile experimental questions. However, the submitted version does not provide the real-LLM evidence needed for its central empirical claims, and several structural arguments and numerical comparisons require correction.

Critical strengths

Treating the output interface and position mapping as explicit experimental variables is useful. Separating calibration from discrimination is also appropriate. The manuscript discloses the mock endpoint and provides a reasonably concrete description of the intended evaluation harness.

Critical weaknesses
  1. The central empirical claims lack real-LLM evidence. The end of Section 4 (pp.5-6) and Appendices K/L disclose that all reported numbers come from a mock endpoint. Appendix L describes a prompt-hash seed, a hidden signal derived from next-day returns, skill parameters standing in for the named backbones, and constructed under-confidence/weak discrimination. Thus the agreement, calibration, routing, and efficiency results do not establish corresponding properties of Qwen models. The disclosure is acknowledged; the concern is the mismatch between this pipeline demonstration and the abstract, model-labelled tables, and conclusions. Real model runs are necessary to support the claimed empirical contribution.

  2. Independent logical and numerical issues undermine interpretation even as a synthetic study. Sharing the action set {-1,0,+1} does not imply sharing a policy: interfaces can select different directions on different days, so gains need not arise only from sizing. The abstract's full-size characterization of L0-L2 also conflicts with the L2 confidence-scaled sizer. Under Section 3.4's additive cost accounting, nontrivial escalation necessarily costs more than always-execute, so requiring non-increased token cost makes the dominance test partly predetermined. Finally, Table 6 reports MSFT L1/L2/L3 Sharpe values 2.69/2.60/1.83, but positive L2/L3-minus-L1 differences of +2.18/+1.20; the displayed differences would instead be approximately -0.09/-0.86. These discrepancies require a corrected argument, a matched-cost routing comparison, and clarification or recomputation of the tables.

Questions for authors
  1. Does the submitted version contain any real-LLM decision logs or results? If not, which central empirical claims are intended to hold independently of the mock's construction?
  2. How are the Table 6 differences calculated, and what precise dominance inequalities are used under additive routing costs? How should the action-set argument be revised to account for different state-to-action policies?
Reviewer_ar8aOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how different output formats affect the decisions of LLM trading agents. It compares four interfaces, from free-text decisions to structured probability distributions, and examines their effects on trading performance, confidence calibration, and confidence-based routing. The evaluation uses the InvestorBench framework and considers three Qwen2.5 model sizes.

Critical strengths

The paper addresses an interesting question about how output formats influence agent decisions. The experimental framework is clearly described, and the distinction between confidence calibration and the ability to identify correct decisions is useful.

Critical weaknesses

The authors explicitly state that all reported results were generated using a deterministic mock endpoint, so the central claims about LLM decision-making, calibration, and trading performance are not supported by the experiments presented. The reported differences between output interfaces, improvements in risk-adjusted performance, and failure of confidence-based routing are based on mock-generated decisions. These findings cannot establish how real LLM trading agents behave. The paper therefore lacks the empirical evidence needed to support its contribution.

Questions for authors

Why were the actual LLM experiments not completed before submission? Can the authors provide results from the proposed Qwen2.5 models to demonstrate that the reported findings hold for real LLM-generated decisions?

Reviewer_bW59OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes a four-level output-interface framework for trading agents and studies position sizing, confidence calibration, and routing. All reported experiments use a deterministic mock endpoint on historical market data.

Critical strengths

Separating output representation, position sizing, and confidence evaluation provides a useful framework for controlled comparisons.

Critical weaknesses
  1. All reported results come from a mock endpoint (§4, Appendix L), whose confidence properties are imposed by construction. These experiments do not validate the main claims about actual LLM behavior, despite their presentation as LLM findings.
  2. Risk reductions accompany substantially reduced exposure. Without an exposure-matched constant-sizing baseline, the added value of probability-based sizing remains unclear; the reported Sharpe improvements are not statistically significant.
Questions for authors

Is there empirical validation that the mock reproduces real LLM behavior, particularly interface sensitivity and confidence discrimination? Are exposure-matched sizing comparisons available?

Reviewer_S1yGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how output formats, position sizing, and confidence-based routing affect trading agents, using a mock simulator in the current version.

Critical strengths

None

Critical weaknesses

It is unfinished: Section DEFGHI is empty.

Section J writes, here I quote,

"Every number in this version was produced by this mock on the real InvestorBench price and news series; the mock’s confidence is, by construction, under-confident and weakly discriminative, so the calibration and routing findings of Sections 6 and 7 describe the mock and must be re-read once vLLM logs replace it."

The writing is confusing.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.