acceptodds
Withdrawn from ICLR 2027

Can AI Make Money in Crypto? Measuring the Gap from Backtests to Real Markets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

AI-based trading methods have rapidly evolved from machine learning and reinforcement learning to large language models (LLMs) and trading agents, yet their performance is still predominantly assessed through historical backtesting. Such evaluations provide limited evidence of whether a method can generalize to unseen future markets or whether its backtested performance can be sustained in realistic trading frictions (e.g., latency, slippage, liquidity constraints, and market impact). We present a unified benchmark that evaluates representative machine learning, reinforcement learning, LLM-based, and agent-based trading methods in cryptocurrency markets through three progressively more realistic stages: historical backtesting, prospective exchange-based paper trading, and real-money live trading. These stages jointly increase temporal realism by moving from historical to unseen future markets, and execution realism by moving from offline simulation toward live trading. This protocol enables us to quantify the backtest-to-realization gap, identify when performance begins to deteriorate, and compare how this gap differs across major classes of AI trading methods. We further provide a unified open-source system supporting all three evaluation stages, together with a public platform that continuously updates benchmark results.

Primary area:
datasets and benchmarks
Keywords:
AI4trading, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1gdeOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the problems faced by the existing AI trading methods in evaluation. On the one hand, the existing AI trading methods are often evaluated in the back test data, lacking the evaluation close to the real scene. On the other hand, the existing evaluation can not put the methods of different methodological families into the same evaluation protocol. In order to solve these two problems, this paper proposes a unified evaluation framework of three levels, puts different types of AI trading methods under the same framework for evaluation, and designs a three-level evaluation method gradually close to the real environment. The authors evaluated a large number of advanced baselines under the same protocol and found that the existing methods still performed poorly in the real environment. Future research needs to further consider the difference between the back test evaluation on the dataset and the real world transaction.

Critical strengths

A three-level evaluation framework gradually approaching the real situation: This paper uses a unified framework to evaluate multiple existing advanced AI trading methods. The three-layer evaluation framework proposed by it, from backtesting to paper trading to real trading, and corresponding experiments, demonstrate how the performance of different methods changes as they gradually approach reality. This is a better perspective for evaluating methods, which can help researchers measure the performance differences between different methods in backtesting and reality, effectively inspiring future research to consider how to design methods that can achieve better trading results in real environments. I think this is the critical strength of this paper.

Critical weaknesses
  1. Unclear contribution: This paper mentions that one of the two core issues addressed in this article is to implement multiple different types of methods for unified evaluation. However, from an innovative perspective, the authors only unified the input and output, without delving deeper into the differences between the models. Therefore, I think the contribution of this work is more reflected in the three-level evaluation framework. The contribution of a unified framework lacks persuasiveness.
  2. Insufficient analysis: Although the three-layer evaluation framework can better demonstrate the performance differences of the model as it gradually approaches the real environment. The authors' analysis in this regard seems insufficient. The authors presented the model performance at each level of evaluation, but did not delve deeper into the differences between each type of method and which methods may be closer to the real environment. The authors mainly attribute the differences to the differences between reality and data backtesting, without delving into how to reduce such differences from the differences in methods, lacking some insights that could have been obtained.
Questions for authors
  1. This paper conducted experiments on three-level situations, demonstrating the performance differences of different methods as they gradually approach the real environment. However, is this difference caused by different market environments? For example, in a paper trading test, it is a bull market, but in a real test, it is a different situation? The author should discuss this matter
  2. How paper trading is conducted and what are the essential differences from the real situation? The author should provide a more in-depth explanation of this
  3. How to use this benchmark in practical situations? Especially in real situations, different market conditions may result in different performance of the model. In the process of evaluating the model, how do the authors think to use this benchmark to obtain effective and stable model evaluation?
  4. Has the paper found any generalizable insights, such as under what circumstances certain methods perform better in real-world situations
Reviewer_LtV2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a cryptocurrency trading benchmark covering 32 AI methods, with historical backtesting, prospective exchange-based paper trading, and real-money deployment. It reports substantial changes in performance across stages and finds that none of the four live-trading methods outperforms Bitcoin buy-and-hold during the evaluated period. The benchmark addresses a relevant deployment problem, but inconsistencies in the experimental protocol limit the conclusions that can be drawn from the reported gaps.

Critical strengths
  1. Comparing historical simulation with prospective and real-money execution addresses a practical weakness of trading research.

  2. The study covers predictive ML, RL, direct LLMs, and agents, and reports execution latency, price deviations, and order-size deviations alongside financial outcomes.

  3. The results show why favorable backtests need prospective validation. The appendix documents prompts and deployment adaptations, and the proposed shared infrastructure could support continued evaluation.

Critical weaknesses
  1. Sections 3.1 and 3.3 state that selected configurations remain fixed across stages. However, Appendix B.3 (pp. 19-20) describes 2025 direct-model backtests on BTC perpetuals at 10x leverage, while prospective execution uses 2x leverage, four assets, and exposure caps. DeepSeek also switches from Pro to Flash within its historical trajectory. These changes confound the claimed temporal generalization gap.

  2. Sections 4.2-4.3 compare identical calendar windows, which is useful, but do not establish matching model snapshots, stochastic decisions, memory states, or initial portfolios. Different decisions and holdings can generate return gaps without execution frictions. Replay recorded decisions under alternative execution models and separate that analysis from end-to-end strategy runs.

  3. Section 3.2 and Appendix B.2 compare ML portfolios selected from ten assets, ETH-only RL policies, and LLM systems with different information, leverage, and short-selling permissions. These design choices can explain family-level rankings and reversals. Include a controlled subset with matched assets and risk constraints, plus ETH and equal-weight portfolio baselines, or restrict conclusions to the specific deployed systems.

  4. Section 3.3 describes long positions as spot trades, whereas Appendix B.3 uses perpetual positions for the direct models. The historical direct-model fee is 0.10%, despite the main text specifying 0.05% for perpetual short trades. Report actual initial live capital, instrument-specific fees, funding and liquidation treatment, and inference costs. Provide the return series and exact alpha/Sharpe definitions, including sampling, annualization, and risk-free rate. These details materially affect both profitability and the claimed reduction in risk-adjusted performance gaps.

Questions for authors
  1. Do the main conclusions persist when historical and prospective evaluations use identical model versions, leverage, asset universes, and exposure limits?

  2. Can a common recorded order stream isolate the effects of fills, fees, and latency from differences in stochastic decisions and agent state?

  3. Do the reported family rankings persist when methods trade the same assets under comparable leverage and exposure limits?

Reviewer_vDSrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a staged benchmark for AI trading methods in cryptocurrency markets. It evaluates 32 methods against a BTC buy-and-hold baseline on a 10-asset universe from OKX. The three stages are: 1) Historical backtest on 2025 data, with training on 2021–23 and validation on 2024. 2) Exchange-hosted paper trading on OKX from 2026-06-04 to 09-21, for 14 selected methods. 3) Real-money live trading from 2026-08-24 to 09-21, for 4 methods.

The author found that 1) ML methods dominate the 2025 backtest, but this ranking largely reverses in paper trading. 2) Backtest and paper trading over the same period still differ materially, by an average of 6.25 pp in cumulative return. These differences don’t correlate with measured execution discrepancies. 3) No live method beats BTC, and all four have negative alpha.

Critical strengths
  1. Separating temporal realism from execution realism is clean. Running backtest and paper trading over the same window is a good design for isolating execution effects.

  2. Few benchmarks place gradient boosting, deep sequence models, HFT RL, and LLM agents in one harness.

  3. Measuring latency, price deviation, and order-value deviation in both paper and live stages is useful. The roughly 6× jump in price deviation from paper to live is a concrete, actionable observation. In addition, the paper reports that nothing beats the market live, rather than spinning the results.

Critical weaknesses
  1. Configurations are not actually held fixed across stages, which confounds the central RQ2 “reversal.” Sec. 3.1/3.3 states that selected configurations are frozen. Howeverm Appendix B.3 says the 2025 DeepSeek/Qwen backtests trade BTC-USDT perpetuals only at 10× leverage, and the DeepSeek run switches model variant mid-trajectory. Prospective trading instead uses 2× leverage, multiple assets, and 15%/60% exposure caps. The LLM side of the “paradigm reversal” therefore compares different strategies, not the same strategy across market regimes. The ML side also chose its timeframe from 5m/15m/1h/4h, and the chosen value is not reported per method.

  2. The same-period backtest-vs-paper gap is not decomposed, and the evidence points to simulator or stochasticity effects rather than execution. Paper beats backtest for 10 of 14 methods. Adding real frictions should usually hurt, so this strongly suggests the backtest fill model is miscalibrated in some way or that decisions differ between runs. In addition, LLM sampling noise is uncontrolled. Each paper-trading result is a single run, and backtests over the paper period aren’t repeated.

  3. Selection is inconsistent and undocumented, and the paper ignores the finance literature on backtest overfitting. Sec. 3.3 says five methods advance to live trading; Sec. 4 and Table 4 say four. Selection is said to be by top paper-trading alpha, yet TRA goes live while MacroHFT and FinGPT do not.

Questions for authors
  1. See in "Critical Strengths"

  2. How much real capital was deployed per method, and what was the order size relative to top-of-book depth?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.