- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies the problems faced by the existing AI trading methods in evaluation. On the one hand, the existing AI trading methods are often evaluated in the back test data, lacking the evaluation close to the real scene. On the other hand, the existing evaluation can not put the methods of different methodological families into the same evaluation protocol. In order to solve these two problems, this paper proposes a unified evaluation framework of three levels, puts different types of AI trading methods under the same framework for evaluation, and designs a three-level evaluation method gradually close to the real environment. The authors evaluated a large number of advanced baselines under the same protocol and found that the existing methods still performed poorly in the real environment. Future research needs to further consider the difference between the back test evaluation on the dataset and the real world transaction.
Critical strengths
A three-level evaluation framework gradually approaching the real situation: This paper uses a unified framework to evaluate multiple existing advanced AI trading methods. The three-layer evaluation framework proposed by it, from backtesting to paper trading to real trading, and corresponding experiments, demonstrate how the performance of different methods changes as they gradually approach reality. This is a better perspective for evaluating methods, which can help researchers measure the performance differences between different methods in backtesting and reality, effectively inspiring future research to consider how to design methods that can achieve better trading results in real environments. I think this is the critical strength of this paper.
Critical weaknesses
- Unclear contribution: This paper mentions that one of the two core issues addressed in this article is to implement multiple different types of methods for unified evaluation. However, from an innovative perspective, the authors only unified the input and output, without delving deeper into the differences between the models. Therefore, I think the contribution of this work is more reflected in the three-level evaluation framework. The contribution of a unified framework lacks persuasiveness.
- Insufficient analysis: Although the three-layer evaluation framework can better demonstrate the performance differences of the model as it gradually approaches the real environment. The authors' analysis in this regard seems insufficient. The authors presented the model performance at each level of evaluation, but did not delve deeper into the differences between each type of method and which methods may be closer to the real environment. The authors mainly attribute the differences to the differences between reality and data backtesting, without delving into how to reduce such differences from the differences in methods, lacking some insights that could have been obtained.
Questions for authors
- This paper conducted experiments on three-level situations, demonstrating the performance differences of different methods as they gradually approach the real environment. However, is this difference caused by different market environments? For example, in a paper trading test, it is a bull market, but in a real test, it is a different situation? The author should discuss this matter
- How paper trading is conducted and what are the essential differences from the real situation? The author should provide a more in-depth explanation of this
- How to use this benchmark in practical situations? Especially in real situations, different market conditions may result in different performance of the model. In the process of evaluating the model, how do the authors think to use this benchmark to obtain effective and stable model evaluation?
- Has the paper found any generalizable insights, such as under what circumstances certain methods perform better in real-world situations