- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces PolicyValue-Bench, a dataset of 5,125 Chinese government policy documents covering 2021–2025, and PolicyState, a framework designed to incorporate policy information into existing trading strategies. The model combines policy text, LLM-generated assessments, market features, and recurrent opportunity and risk states to determine exposure adjustments and cash exits. The authors evaluate the approach across five Chinese sector ETFs and four underlying trading strategies. The main results cover 2025, with an additional expanding-window evaluation for 2024. PolicyState reportedly achieves an average net cumulative return of 16.25%, compared with 6.76% for the underlying strategies, alongside improvements in Sharpe ratio and maximum drawdown.
Critical strengths
Relevant problem and potentially useful dataset: The paper addresses an interesting question, namely, whether government policy announcements contain incremental trading information beyond what is already reflected in market prices. The construction of a dataset linking policy documents, historical announcements, and sector ETF returns is a useful contribution.
Reasonably comprehensive model comparisons and ablations: The authors compare their approach against several text-based models, LLMs, and financial tradnig agents, as well as a more directly comparable GRU architecture. The ablation studies also help assess the contributions of the different components. These are useful experiments, although they do not fully resolve the concerns about the financial evaluation.
Critical weaknesses
Lack of volatility targeting and risk-normalized performance comparisons: My main concern is the absence of a proper risk-normalized evaluation. The authors report substantial improvements in cumulative returns and maximum drawdowns, but do not report the realized annualized volatility of the individual strategies or compare their performance under a common volatility target. This is a significant omission for a paper evaluating financial trading strategies. Without knowing the volatility and risk exposure of each method, it is difficult to interpret the differences in cumulative returns and maximum drawdowns. A strategy generating higher returns may simply be taking more risk, while lower drawdowns may reflect lower exposure rather than better investment decisions. The reported Sharpe and Sortino ratios provide some risk-adjusted information, but their statistical uncertainty is not reported. In particular, there are no confidence intervals or significance tests for the differences in Sharpe, Sortino, or maximum drawdown against the competing methods. Given that the paper's main claims are based on improvements in trading performance and risk management, I consider this a fundamental weakness of the experimental design. The authors should report realized annualized volatility, average exposure, and performance under comparable risk budgets, rather than relying primarily on raw cumulative returns and drawdowns.
Limited out-of-sample evaluation and insufficient evidence of statistical robustness: The main experimental results are based on a single out-of-sample year, 2025. Although Appendix D includes an expanding-window evaluation for 2024, this amounts to only two annual test periods, which is limited for a financial strategy intended to operate under changing market conditions. A more extensive walk-forward evaluation would be needed to assess whether the reported improvements are consistent across different market regimes, particularly periods of elevated volatility, market stress, and changing policy conditions.
Furthermore, the 20 sector-strategy combinations should not be interpreted as 20 independent tests. The four strategies within each sector trade the same underlying ETF and are exposed to the same market movements and policy announcements. The authors provide a bootstrap confidence interval for the cumulative return improvement over the matched base strategies, which is useful. However, the reported p-value of 0.0382 is relatively close to the conventional significance threshold, and comparable uncertainty estimates are missing for the risk-adjusted metrics and the performance difference against the strongest architectural baseline. Taken together, these limitations make it difficult to assess how robust the reported improvements really are.
Questions for authors
Volatility Targeting: Why have the authors not reported realized annualized volatility for each strategy? Could they provide a comparison of all methods under a common volatility target or risk budget, including returns, Sharpe ratios, and maximum drawdowns?
Confidence Intervals: Could the authors provide confidence intervals or statistical significance tests for teh differences in Sharpe ratios, Sortino ratios, and maximum drawdowns, particularly against the full-input GRU baseline?
Out-of-sample performance: Why is the out-of-sample evaluation limited to 2024 and 2025? Could the authors provide a more extensive walk-forward analysis, or at least a breakdown of performance across different market and volatility regimes?
Impact of policy signals: Could the authors clarify how much of the reported improvement comes from policy-driven trading signals rather than changes in average market exposure or the model's ability to exit positions during market declines?