- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes an output-interface ladder for trading agents, from free text to typed probability vectors, and studies position sizing, confidence calibration, and escalation routing. These are worthwhile experimental questions. However, the submitted version does not provide the real-LLM evidence needed for its central empirical claims, and several structural arguments and numerical comparisons require correction.
Critical strengths
Treating the output interface and position mapping as explicit experimental variables is useful. Separating calibration from discrimination is also appropriate. The manuscript discloses the mock endpoint and provides a reasonably concrete description of the intended evaluation harness.
Critical weaknesses
-
The central empirical claims lack real-LLM evidence. The end of Section 4 (pp.5-6) and Appendices K/L disclose that all reported numbers come from a mock endpoint. Appendix L describes a prompt-hash seed, a hidden signal derived from next-day returns, skill parameters standing in for the named backbones, and constructed under-confidence/weak discrimination. Thus the agreement, calibration, routing, and efficiency results do not establish corresponding properties of Qwen models. The disclosure is acknowledged; the concern is the mismatch between this pipeline demonstration and the abstract, model-labelled tables, and conclusions. Real model runs are necessary to support the claimed empirical contribution.
-
Independent logical and numerical issues undermine interpretation even as a synthetic study. Sharing the action set {-1,0,+1} does not imply sharing a policy: interfaces can select different directions on different days, so gains need not arise only from sizing. The abstract's full-size characterization of L0-L2 also conflicts with the L2 confidence-scaled sizer. Under Section 3.4's additive cost accounting, nontrivial escalation necessarily costs more than always-execute, so requiring non-increased token cost makes the dominance test partly predetermined. Finally, Table 6 reports MSFT L1/L2/L3 Sharpe values 2.69/2.60/1.83, but positive L2/L3-minus-L1 differences of +2.18/+1.20; the displayed differences would instead be approximately -0.09/-0.86. These discrepancies require a corrected argument, a matched-cost routing comparison, and clarification or recomputation of the tables.
Questions for authors
- Does the submitted version contain any real-LLM decision logs or results? If not, which central empirical claims are intended to hold independently of the mock's construction?
- How are the Table 6 differences calculated, and what precise dominance inequalities are used under additive routing costs? How should the action-set argument be revised to account for different state-to-action policies?