- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes KTD-FIN, a Qlib-based trading benchmark on the CSI300 universe. It has two components: data-layer masking of tickers and dates (bright, stock-blind, date-blind, blinded), and a Barra-style attribution that splits returns into common, style and selection alpha. In the main experiment, ten LLMs trade under blinded × open-research from 2024-01 to 2026-04 and are compared against 18 Qlib baselines and CSI300. The authors report that masking shifts rationales from company narratives to factor reasoning, and that LLM returns are mostly explained by common and style exposure, with negative selection alpha for 9 of 10 models.
Critical strengths
- The two problems it targets, memorisation in backtests and raw return as a proxy for skill, are important.
- The execution rules are realistic, and model coverage is broad.
- The paper is very easy to follow.
Critical weaknesses
-
Prior knowledge is not the same as look-ahead bias. Look-ahead bias means knowledge of realised future outcomes. Statements such as "Moutai, liquor leader" (Appendix A) could actually be point-in-time public knowledge that human investors can also use. Table 17 even finds such statements after the model's knowledge cutoff, where look-ahead is impossible. In this case, full masking also changes the task into price-only technical analysis.
-
Some results are inconsistent. Table 5 and Table 6 are different in both directions: GPT-5.5 is +61.3% by account vs. +36.9% by attribution, DeepSeek +1.4% vs. +18.6%. The effect of masking on returns flips sign across models (Table 2 vs. Tables 3 and 12).
-
No pre-/post-cutoff test. I think a cleaner test is to compare the bright and blinded gap before vs. after each model's knowledge cutoff, across several models. The paper has one model and one 43-day post-cutoff window, and does not report any model's cutoff. 43 days is also considered to be too short from my perspective. I would suggest using earlier models to have a larger post-cutoff testing period.
-
Minor: missing many related works. This field have been explored pretty much but many of them are not cited and discussed in the related work and for some of these existing studies, it would be valuable to explain how this paper builds upon them.
Assessing Look-Ahead Bias in Stock Return Predictions Generated by GPT Sentiment Analysis, JFDS 2024 Time Machine GPT, Findings of NAACL 2024 Lookahead Bias in Pretrained Language Models, ICML 2025 The Memorization Problem: Can We Trust LLMs' Economic Forecasts?, arXiv 2025 Can LLM-based Financial Investing Strategies Outperform the Market in Long Run?, KDD 2026 Look-Ahead-Bench: A Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance, arXiv 2026 When Agents Trade: Live Multi-Market Trading Arena for LLM Agents, WWW 2026 Summoning the Oracle to Slay It: Mitigating Look-Ahead Bias in Financial Backtesting with Large Language Models, EMNLP 2026 Evaluating LLMs in Finance Requires Explicit Bias Consideration, ICML 2026
Questions for authors
See the weakness section. My main question is basically: How would you decouple look-ahead bias and prior knowledge?
3 comments
Thanks for the rebuttal. I will raise my overall assessment to 3 given that the requested experiment is added.
Thank you for revisiting our work and for your helpful suggestions.
Thank you for recognizing the research question, execution rules, and clarity of the paper, and for identifying where the return statistics and temporal comparisons need clearer presentation.
W1 / Q1. Prior knowledge and look-ahead bias. We agree that a publicly known fact such as “Moutai is a liquor leader” differs from knowledge of subsequently realized returns. Our leakage concern is the latter: memorized outcomes from the backtest period. Stock identities and dates provide cues through which either type of knowledge can enter an agent's decisions. KTD-FIN holds market observations fixed while varying access to these cues throughout sequential trading. W10's explanation audit measures identity-conditioned company references after the anchor model's stated cutoff. The revised Introduction and §4.4 clarify this distinction.
W2. Return definitions and masking effects. Tables 5 and 6 report different statistics:
For example, two daily returns of +10% compound to +21% but sum to +20%. Their difference depends on the return path. In the seed-level examples, GPT-5.5 has a compounded account return of 61.3% and a daily account-return sum of 55.0%; DeepSeek has corresponding values of 1.4% and 11.2%. Cross-seed means and medians introduce a separate aggregation difference. Section 4.1 and Appendix S now define these conventions, and Table 6 labels “Port” as “Add. sum” to distinguish it from compounded account return.
Tables 2, 3, and 12 cover different models and window collections; the anchor uses five seeds and the additional model comparisons use three. Masking tests identifier dependence, whose return effect varies across models and market windows. We distinguish this from the behavioral evidence: across 30 matched model/mode/window comparisons, mean absolute changes are 18.7 percentage points in capital utilization and 15.7 points in abstention (§4.2). Return comparisons are reported by model and window, while the explanation audit directly examines the information cited in trading decisions.
W3. Temporal comparisons across models. Thank you for recommending a broader temporal comparison. We expanded the ''doubao-seed-1-6-251015'' evaluation to eight windows, including a longer window from 2025-11-03 to 2026-02-02. Using the version's October 2025 release as the temporal reference, Appendix R, Table 23 now reports six pre-release and two post-release windows:
We also disaggregated the Step-3.5-Flash, GPT-5.4-mini, and hy3 experiments in Tables 2 and 3 into pre- and post-cutoff periods. Across these three models, bright’s return advantage generally diminishes after the cutoff.
W4. Related work. Thank you for these references. We have added the suggested work on temporal isolation, point-in-time models, look-ahead detection, and live-market evaluation. Related Work now explains how KTD-FIN builds on these directions through consistent information control across sequential agent–tool interactions and return attribution.