- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces AlphaOpsBench, which evaluates whether LLMs can turn a rough prediction-market trading idea into an executable strategy program that stays faithful to the original intent.
Benchmark details: The data covers 3 months of Polymarket. The tasks consist of 581 strategy records compiled into 477 tasks, plus 36 controlled diagnostic tasks. The evaluation compares Direct generation with Staged generation (design first, then code), and replays strategies on two engines: V3 (based on trade bands) and PML2 (based on the L2 order book).
Only one model is evaluated, Qwen3.8-27B. Across 5 repeated runs, Direct passes 35/180 controlled tasks and Staged passes 20/180. On the 24 real-world strategies, the pass rate is 0/240.
Critical strengths
Meaningful problem, sound framing. The benchmark evaluates strategy implementation, and it scores faithfulness separately from executability. Prediction markets are well suited to this kind of evaluation. Furthermore, the paper builds two replay engines on Polymarket trade, limit order book (LOB), and settlement data, and this infrastructure could be reused beyond this paper.
Critical weaknesses
It is unclear whether the framework actually measures an LLM's ability to implement strategies. The main text gives no operational definition of what counts as a canonical pass, and there is no agreement check against human judgment. Only one model is evaluated, and thinking mode seems to be disabled (Appendix J). In addition, the reported financial results do not seem able to distinguish good strategies from bad ones, and no baselines (e.g. random or buy-and-hold strategies) are reported.
Questions for authors
- What is a canonical pass? Please give a precise definition with concrete worked cases. Did the authors measure how well the canonical checker agrees with human or expert judgment?
- How do other models perform on this benchmark, both with and without thinking mode enabled?
- In the final reported results, which numbers come from raw LLM outputs, and which come from outputs that were adjusted or post-processed?