acceptodds
Under review as a conference paper at ICLR 2027

AlphaOpsBench: Benchmarking End-to-End Alpha Strategy Operationalization in Prediction Markets

Abstract

Large language models increasingly generate quantitative trading strategies, yet existing benchmarks assume standardized assets, numerical features, or directly compilable strategy representations—assumptions that prediction-market strategies violate, since a coarse idea may leave the traded outcome, causal information source, signal definition, threshold, sizing, order policy, exit, and settlement behavior unspecified. We introduce AlphaOpsBench, which evaluates end-to-end operationalization from source-grounded economic hypotheses to auditable executable programs over 581 source-preserving strategy records and a lifecycle-scale Polymarket dataset with 1.28 million binary markets, 183.6 million cleaned executions, settlement evidence, and limit-order-book history, comparing Direct generation against a Staged design-then-code protocol. In a corrected independent-generation study over 36 controlled tasks and 24 preregistered real strategies, strict end-to-end validity remains rare: Direct and Staged obtain 35/180 and 20/180 canonical passes on the controlled cohort and no confirmed pass on the real cohort, and repeated generations vary substantially in model-owned economic choices. By contrast, 775,725 of 783,655 scheduled historical replays complete, showing that replayability is a far weaker property than source-faithful operationalization. Financial outcomes depend on the declared execution model and available historical evidence, and fee and liquidity experiments show that execution costs alter subsequent trading paths rather than acting only as ex-post deductions. AlphaOpsBench thus separates strategy fidelity, behavioral validity, historical executability, and financial performance in an evidence-aware benchmark for LLM-based quantitative research in prediction markets.

Primary area:
datasets and benchmarks
Keywords:
Prediction Markets, Large Language Models, Alpha Strategies, Quantitative Finance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_yAfHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AlphaOpsBench, which evaluates whether LLMs can turn a rough prediction-market trading idea into an executable strategy program that stays faithful to the original intent.

Benchmark details: The data covers 3 months of Polymarket. The tasks consist of 581 strategy records compiled into 477 tasks, plus 36 controlled diagnostic tasks. The evaluation compares Direct generation with Staged generation (design first, then code), and replays strategies on two engines: V3 (based on trade bands) and PML2 (based on the L2 order book).

Only one model is evaluated, Qwen3.8-27B. Across 5 repeated runs, Direct passes 35/180 controlled tasks and Staged passes 20/180. On the 24 real-world strategies, the pass rate is 0/240.

Critical strengths

Meaningful problem, sound framing. The benchmark evaluates strategy implementation, and it scores faithfulness separately from executability. Prediction markets are well suited to this kind of evaluation. Furthermore, the paper builds two replay engines on Polymarket trade, limit order book (LOB), and settlement data, and this infrastructure could be reused beyond this paper.

Critical weaknesses

It is unclear whether the framework actually measures an LLM's ability to implement strategies. The main text gives no operational definition of what counts as a canonical pass, and there is no agreement check against human judgment. Only one model is evaluated, and thinking mode seems to be disabled (Appendix J). In addition, the reported financial results do not seem able to distinguish good strategies from bad ones, and no baselines (e.g. random or buy-and-hold strategies) are reported.

Questions for authors
  1. What is a canonical pass? Please give a precise definition with concrete worked cases. Did the authors measure how well the canonical checker agrees with human or expert judgment?
  2. How do other models perform on this benchmark, both with and without thinking mode enabled?
  3. In the final reported results, which numbers come from raw LLM outputs, and which come from outputs that were adjusted or post-processed?
Reviewer_p26uOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AlphaOpsBench to evaluate whether LLMs can translate incomplete prediction-market trading ideas into executable strategies faithful to their original intent. It then evaluates strategy fidelity, behavioral validity, historical executability, and financial performance, comparing Direct generation with a Staged design-then-code workflow. The findings shows that successfully running a program or completing a backtest does not necessarily mean that the original strategy has been implemented correctly. And backtest outcomes depend on execution model, fees and liquidity constraints. The framework helps identify failures across the strategy development process.

Critical strengths

The paper addresses an interesting problem in prediction markets. The benchmark separately evaluates fidelity to the source strategy, consistency between the generated design and its implementation, executability, and financial performance. By recoding source requirements and model-owned decisions, it makes strategy fidelity easier to assess and help identify failures.

Critical weaknesses
  1. The low pass rate at current benchmark should be interpret better. Although the paper includes tests within reference implementations and faulty variants, it does not exactly means how reliably the final evaluator distinguishes known correct implementations from incorrect ones. Therefore, it still remains how much the low baseline score reflect model limitations versus evaluator limitations or insufficient verification evidence. More evidence of the reliability of final evaluator and clearer criteria for classifying failure would be better.

  2. The large-scale replays show that the research packages can run, but these include recovered or modified version of the original programs. The paper should explain which changes address formatting or interfaces and which alter strategy behavior. Since no candidate in the original 477-task cohort achieved a canonical pass, completed replays alone do not establish faithful strategy execution. Linking each replay candidate to its original version would make these results easier to interpret.

Questions for authors
  1. Could author clarify how canonical PASS, FAIL, and UNVERIFIED are determined and validated? Figure 8 reports zero own-policy consistency across all six groups, while the controlled experiments report 55 canonical passes. How do the evaluated candidates and criteria differ between these analyses? Result on known-correct implementations and faulty mutations would also help assess the final evaluator's reliability.

  2. How does the 456 replay packages map to the original programs? Please summarize recovery steps, repairs, and semantic changes, and report results separately for programs whose strategy semantics were preserved.

  3. Can validated reference implementations pass canonical checks and execute intended trades when sufficient data and triggering conditions are present?

  4. How do pass rates and failure types change with reasoning enable or stronger models?

Reviewer_YJmYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AlphaOpsBench, a benchmark for evaluating the end-to-end operationalization of underspecified trading strategies in prediction markets. The benchmark combines 581 source-preserving strategy records with Polymarket market, execution, settlement, and order-book data. It evaluates both direct strategy-and-code generation and a staged design-then-code pipeline, using source-fidelity checks, behavioral validation, historical replay, and execution-cost analyses.

The paper reports low strict end-to-end validity: in the independent-generation study, Direct and Staged achieve 35/180 and 20/180 canonical passes on controlled tasks, respectively, and neither achieves a confirmed pass on the 24 real-strategy tasks. It also reports 775,725 completed historical replays across the two replay frameworks. The paper's central claim is that executable code and successful replay are insufficient evidence of faithful strategy operationalization.

Critical strengths
  1. Prediction-market strategies require reasoning about outcome semantics, information timing, order execution, position transitions, and settlement. These aspects are not adequately captured by code executability alone.

  2. The paper distinguishes source fidelity, behavioral validity, historical executability, and financial performance instead of collapsing them into a single return metric. This is a sensible design principle.

  3. The paper explicitly accounts for partial fills, liquidity constraints, fees, and settlement evidence, and acknowledges that historical replay is not equivalent to live trading. This is preferable to presenting backtest returns as definitive evidence of strategy quality.

Critical weaknesses
  1. Canonical acceptance is the key outcome used to support the claim that faithful operationalization remains difficult. Yet the very low pass rate on controlled tasks and the complete absence of confirmed passes on real strategies raise an important ambiguity: do these results reveal model limitations, or do they partly reflect the benchmark's specification and validation choices? The paper does not provide sufficiently convincing independent evidence to resolve this ambiguity. In particular, the real-strategy results are difficult to interpret when no evaluated generation passes the full criterion.

  2. The main comparison evaluates Direct and Staged generation using only Qwen3.8-27B. A benchmark paper should demonstrate that its tasks and metrics provide useful comparisons across multiple capable models or model families. Without such comparisons, the paper establishes that one model struggles under one evaluation setup, but does not establish the benchmark's broader discriminative value. The paper's extensive infrastructure and data collection make this limitation particularly consequential.

  3. The 775,725 completed replays are a prominent result, but replay completion is distinct from semantic fidelity and behavioral correctness. The paper itself acknowledges that the replay population is not equivalent to the canonical acceptance population. Therefore, the replay count should not be treated as validation of the benchmark's core claim. The financial results are additionally conditional on different candidate populations and execution models, limiting their comparability and interpretation.

  4. The paper argues that existing financial benchmarks do not adequately evaluate the operationalization of underspecified prediction-market strategies. This is plausible, but the paper would be stronger with a direct comparison showing what existing benchmarks or evaluation protocols miss, and whether AlphaOpsBench identifies failures that those alternatives cannot detect. The conceptual distinction alone is not enough to establish the practical advantage of the proposed benchmark.

Questions for authors
  1. How was the canonical acceptance criterion independently validated? Were generated strategies assessed by domain experts or independent annotators, and what is the agreement between those assessments and the automated checks?

  2. Given that neither Direct nor Staged achieves a confirmed pass on any of the 24 real-strategy tasks, what evidence indicates that this outcome reflects model limitations rather than overly restrictive or incomplete reference contracts?

  3. Why is the main model evaluation limited to Qwen3.8-27B? Can the authors report results for multiple model families, including stronger general-purpose models, to demonstrate that AlphaOpsBench can meaningfully distinguish capability levels?

  4. Can the authors compare AlphaOpsBench with existing prediction-market or executable-strategy benchmarks under a common evaluation protocol, showing which failure modes are uniquely identified by the proposed benchmark?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.