acceptodds
Under review as a conference paper at ICLR 2027

Backtest Mirage: Decision-Aware Auditing of Financial Machine Learning

Abstract

Small errors in a model's scores can change the economic conclusion of a financial backtest. A small change near the selection cutoff can replace a stock, while a much larger change elsewhere leaves the portfolio unchanged. We introduce decision-aware economic auditing: rank candidate interventions by the investment decisions they change, then measure economic impact through paired portfolio continuations. Across all 24 future-input conditions, exposure near the cutoff produces smaller score changes but larger economic deviations than exposure farther away. On CSI 300 and CSI 500 with Ridge and LightGBM, our learned Decision priority captures 34% more impact than a learned score-only baseline at the same 10% checking budget in a frozen 2025 test. In a secondary rank-weighted evaluation, exploratory Allocation captures impact from weight changes among the same stocks. Decision information thus directs a fixed checking budget toward errors with larger effects on the economic evidence from backtests. Code is available at https://anonymous.4open.science/r/Backtest-Mirage-0220.

Primary area:
learning on time series and dynamical systems
Keywords:
financial machine learning, backtest reliability, economic auditing, data leakage, decision-aware evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rNjROpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes decision-aware economic auditing for financial machine learning backtests. Instead of prioritizing potential evaluation errors solely by prediction-score changes, the approach considers their effects on stock selection and portfolio allocation. The authors introduce model-free Replacement and Allocation priorities, together with a learned Decision priority incorporating decision-related descriptors. Economic impact is measured through paired portfolio continuations under identical initial account states and market conditions. Experiments using Ridge and LightGBM on CSI 300 and CSI 500 examine information leakage, execution assumptions, and audit prioritization. In a frozen 2025 evaluation, Decision achieves 30.24% impact capture at a 10% checking budget, compared with 22.56% for Score-only and 29.04% for Replacement. The authors conclude that decision information improves the allocation of limited auditing resources toward economically consequential evaluation errors.

Critical strengths

S1. Relevant Problem Formulation. The paper highlights an important distinction between prediction-score deviations and their downstream economic consequences. Incorporating investment decisions into audit prioritization is a meaningful perspective for evaluating financial machine learning systems (Sections 1–3).

S2. Careful Experimental Organization. The study includes controlled information-exposure experiments, paired portfolio continuations, temporally separated evaluation, and uncertainty estimates (Sections 4–5). The detailed execution and reproducibility specifications provide useful evidence within the controlled experimental setting.

Critical weaknesses

W1. Limited Incremental Contribution Beyond Simple Decision-Based Heuristics (Fundamental). The main contribution emphasizes a 34% relative improvement over Score-only. However, Table 1 shows that the model-free Replacement baseline already achieves 29.04% impact capture, compared with 30.24% for the learned Decision method. The actual incremental gain over this stronger baseline is only 1.19 percentage points. Moreover, this comparison was conducted post hoc, while the corresponding development-period improvement is merely 0.34 points with a confidence interval of [-0.17, 0.84]. Adding account-state features also yields no clearly established additional benefit (Section 5.3). Although the 2025 improvement over Replacement has a positive confidence interval, its practical significance remains limited. The results suggest that most of the demonstrated advantage arises from incorporating straightforward stock-selection changes rather than from the proposed learning mechanism. The paper does not convincingly establish sufficient methodological novelty or incremental value.

W2. Strong Dependence on Predefined Interventions and Clean Counterparts (Fundamental). The framework assumes that candidate interventions and their corresponding clean and altered predictions are already available (Sections 2.2 and 5.1). Appendix M explicitly acknowledges that the method prioritizes supplied interventions rather than discovering unknown pipeline violations. This assumption substantially simplifies the auditing problem, since identifying suspicious operations, constructing valid counterfactual inputs, and obtaining reliable clean references are often central difficulties in real financial backtest auditing. Consequently, the proposed framework addresses a restricted prioritization task rather than the broader detection and validation of unreliable backtests. The paper should more clearly establish how its assumptions can be satisfied in realistic auditing workflows.

W3. Weak Alignment Between the Audit Objective and Final Economic Conclusions (Fundamental). The economic impact metric is defined as the mean absolute daily return difference over a six-session paired continuation (Section 2.2, Eq. 2). While this measures local portfolio deviations, it does not directly evaluate whether an intervention changes the final scientific conclusion of a backtest, such as profitability, Sharpe ratio, maximum drawdown, or relative model ranking. Absolute deviations may also prioritize temporary fluctuations that have limited implications for the overall evaluation. Therefore, improved impact capture does not necessarily demonstrate improved identification of economically misleading backtests. Additional evaluation against conclusion-level distortions is needed to validate the relevance of the proposed auditing objective.

W4. Insufficient Generalization Across Portfolio Policies and Financial Settings (Fundamental). The primary evaluation is restricted to two related Chinese equity universes, Ridge and LightGBM forecasters, and top-decile portfolios with five-session holding periods. The candidate interventions are generated from a limited set of predefined future-information substitutions (Section 5.1). More importantly, the secondary rank-weighted evaluation reveals substantial sensitivity to portfolio construction: the model-free Allocation priority achieves 31.99% capture, whereas the frozen Decision priority achieves only 26.36% (Figure 6, Appendix N). Although this analysis is exploratory, it indicates that the learned priority does not consistently outperform simpler policy-specific heuristics when allocation rules change. Stronger cross-policy and cross-market evaluations are required to support general applicability.

W5. Incomplete Accounting of Practical Auditing Costs (Addressable, but Consequential). The evaluation defines the checking budget primarily as a fixed fraction of audit cases receiving paired portfolio continuations (Section 2.3). However, real auditing also requires generating candidate interventions, obtaining clean and altered predictions, constructing diagnostic features, and acquiring historical responses for fitting the learned priority. Appendix N provides useful timing measurements but explicitly excludes several of these upstream costs, including forecasting, historical label acquisition, and human investigation. Consequently, an identical 10% checking budget does not establish equal end-to-end resource requirements or cost-effectiveness. Given the small improvement over Replacement, the practical justification for the learned method remains insufficient.

Questions for authors
  1. Incremental novelty: Given the small Decision–Replacement gain in Table 1 and the inconclusive development-period comparison, what evidence justifies the additional learned prioritization model over the simpler heuristic?

  2. Realistic auditing: How would the framework operate when potential violations and their clean counterparts are not known beforehand? What assumptions and costs are required to construct the supplied audit cases?

  3. Economic validity: How strongly does the six-session absolute-return-deviation metric correlate with changes in final backtest conclusions, including profitability, risk-adjusted performance, and strategy rankings?

  4. Policy generalization: How do the authors explain the substantial advantage of the model-free Allocation baseline over the frozen Decision model under rank-weighted portfolios? What does this imply about transfer across portfolio construction rules?

  5. End-to-end efficiency: How would including intervention generation, diagnostic construction, historical fitting, and verification costs affect the comparison between Decision and Replacement under an equal computational or monetary budget?

3 comments
AuthorsOpenReview ↗

Follow-up on final economic outcomes and the audit contribution

Dear Reviewer rNjR,

Thank you again for your detailed questions. We are following up on our responses, which add evidence on final economic outcomes, the learned increment, and practical costs.

We extended all 34,984 original cases under both allocation rules to the end of the evaluation window. Without refitting, Decision exceeds Score-only and Replacement in capture of final-return, Sharpe, and maximum-drawdown changes under equal weights.

At the original checking quota, Decision also finds 143 more high-impact cases than Replacement. Our responses explain Allocation's advantage under rank weights and provide measured costs and a concrete audit workflow.

These additions connect the audit task to final economic evaluation and clarify when each priority is useful. We would appreciate your updated assessment of the contribution and welcome further discussion.

Best regards,

The Authors

AuthorsOpenReview ↗

Gentle follow-up on our response

Dear Reviewer rNjR,

We are following up on our responses and the additional evidence on final economic outcomes, the learned increment, and audit costs.

We would appreciate your assessment of these results and their implications for the contribution. We welcome any further questions.

Thank you for your time.

Best regards,

The Authors

AuthorsOpenReview ↗

Request for Feedback on Our Additional Evidence

Dear Reviewer rNjR,

Thank you again for your detailed and thoughtful review. We have provided additional experiments and analyses addressing your concerns, particularly regarding economic validity, methodological contribution, practical costs, and generalizability.

We understand that some limitations remain, and we appreciate your critical perspective. We would be grateful if you could share your thoughts on the new evidence and whether it changes your overall assessment.

Thank you for your time and consideration.

Best regards,

The Authors

Reviewer_8HmgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces decision-aware economic auditing for financial ML backtests: when only some candidate interventions can be checked by paired portfolio replay, prioritize those that change investment decisions, since small score changes near the selection cutoff can matter more than large distant ones. It offers model-free priorities (stock replacement, weight change) and one learned from past checks. In a frozen temporal test on Chinese equities, the learned priority recovers more economic impact than a score-only regression.

Critical strengths
  1. The work has a new and well-posed problem framing. Instead of measuring how much each leakage type inflates performance, the paper asks which candidate errors to verify first under a fixed budget, with a clear objective (impact captured) and baselines from random to oracle (Table 1).
  2. The paper presents creditable evaluation. The primary contrast and success criterion were fixed before the 2025 outcomes were accessed (App. K), the interval rule was calibrated by null simulation (App. F.4), and unfavorable results are reported, including model-free baselines that rival the learned priority (Secs. 5.3–5.4). This makes the reported gains trustworthy.
Critical weaknesses
  1. The learned method adds little over a simple baseline. The headline gain is measured against a score-only regression that cannot see the selection cutoff. Against the paper’s own model-free Replacement baseline, the learned priority’s gain is small and not significant in development, and under rank weights, the model-free Allocation outperforms it. The evidence thus favors a simple diagnostic over the learned priority, which is the paper’s ML contribution.
  2. From my perspective, the motivation is weak. The paper’s premise is that economic checks are too costly to run on every candidate, yet in its own setting, checking all candidates takes only minutes, so the practical benefit of prioritizing is not demonstrated. Also, all evidence comes from one market and from forecasters that lose money even without errors, so it is unclear whether the findings transfer to profitable models or to portfolio rules without a top-K cutoff.

To conclude, my key consideration for the final rating is its weak or, at best, marginal contribution.

Questions for authors

Questions 1. I tried to reproduce your results by accessing the anonymous link. The released scripts run and reproduce the main results from the daily aggregates, but case-level data are not released. I am not sure if I miss it or you did not provide it.

Question 2. Eq. (2) averages absolute daily return differences, so differences that cancel over the six sessions still count as impact. For example, daily differences of +30, −30, +30, −30, +30, −30 bp give an impact of 30 bp/day but leave the backtest’s return unchanged, whereas +5 bp every day gives only 5 bp/day yet raises the return by 30 bp. Since a backtest’s conclusion depends on the signed difference, do the 2025 results still hold if impact is the absolute cumulative difference?

5 comments
AuthorsOpenReview ↗

Follow-up on cumulative-return tests and released case-level data

Dear Reviewer 8Hmg,

Thank you again for checking our code and raising the daily-cancellation example. We are following up on our two responses.

Using the absolute cumulative-return difference from your example, Decision captures 29.57%, versus 28.26% for Replacement and 21.75% for Score-only, at the same check count. The gain over Replacement is 1.31 pp [0.09, 2.52]. The benefit also holds for twenty-session terminal wealth.

The anonymous repository now includes case-level data and verification scripts. Our second response also clarifies practical costs and reports positive reference returns in the 2025 evaluation window.

We would appreciate your updated assessment in light of this evidence and welcome further questions.

Best regards,

The Authors

Thank you for the response and for releasing the case-level data. My questions, for that part, are resolved. My main concern, the motivation, remains. The paper assumes that economic checks are too costly to run on every candidate, yet in its own setting checking all candidates takes under four minutes of aggregate work, referred to Appendix N.3. By the paper's own cost model, forecasting is shared preparation that every priority also needs, so prioritization saves only the omitted replay work, a few minutes here. Moreover, the 2025 accounts earn 15–31%, but 2025 was a strongly rising year for Chinese equities, so absolute returns do not show that the forecasters have skill; returns relative to the market benchmark would. I therefore keep my score.

AuthorsOpenReview ↗

Clarification on audit validity, forecast skill, and replay costs

Dear Reviewer 8Hmg,

Thank you for reviewing our responses and confirming that the data-related questions are resolved. We would like to clarify the audit objective and its relation to the replay timing.

The forecasters are the systems being audited.

Our method chooses which supplied candidate errors to check first, based on their economic consequences. We evaluate audit quality through the impact found by these checks and its connection to final backtest outcomes.

Market-relative returns assess forecast skill. The positive account returns we reported describe the tested accounts. The audit asks a separate question: how much does a pipeline error distort the evidence used to judge a model?

Section 4.2 gives a direct example. With All19 features, joint training-and-scoring exposure turns negative reference returns into apparent profitability. Such an error can lead a researcher to accept a misleading result. Errors can also lower reported returns and lead to an incorrect rejection. Audit validity therefore matters for both profitable and loss-making strategies.

What the replay timing establishes.

Appendix N.3 reports 231.18 seconds of aggregate work for all rank-weighted continuations. Forecasting is shared preparation, so the computational saving from checking fewer cases comes from the omitted replay work. Exhaustive replay is practical in this lightweight simulator.

This complete replay gives us the reference needed to measure which checking orders find larger economic effects earlier. At 10% of checks, Allocation recovers 31.99% of total impact without fitting. The measured benefit is more impact found per check.

The learned increment also survives stricter case construction. After allocation merging and zero-impact screening, Decision improves over Replacement by 1.54 pp [0.80, 2.28] at the same 3,482 checks. For twenty-session terminal wealth, the gain is 2.04 pp [0.41, 3.66].

These findings show how decision information improves economic checking through both simple and learned priorities. The contribution combines an economic audit task, evidence on how pipeline errors affect investment outcomes, and methods that find more impact at the same check count. We would appreciate your reassessment in light of these findings.

Best regards,

The Authors

Reviewer_drjTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to prioritize supplied backtest interventions under a fixed budget of paired portfolio checks. It compares score-based diagnostics with selected-stock replacement, allocation distance, and a learned decision-aware priority. In a frozen 2025 evaluation, Decision captures 30.24% of aggregate impact at a 10% checking budget, versus 22.56% for the learned score-only baseline. Additional experiments examine information timing, cutoff proximity, account state, and rank-weighted allocation.

Critical strengths

The distinction between prediction changes and decision consequences is useful. The paired continuation design preserves the initial account state and execution constraints, and the main candidate masks are independent of the selection boundary. The paper provides extensive controls and carefully distinguishes the primary temporal comparison from post-hoc and exploratory analyses. Reporting the small additional value of account-state features and the strong performance of simple decision-based rules is particularly helpful.

Critical weaknesses
  1. The economic-equivalence control is consequential to the main budget claim. The primary portfolio is equally weighted, but cases are deduplicated by ordered selection (Section 2.2). Appendix N.3 reports 18,460 membership-preserving cases in a pool of 34,984. Under the stated equal-weight policy, these appear analytically identifiable as zero-impact cases without portfolio continuation; different ordered selections can also induce identical allocations. Please compare methods after merging identical target allocations and screening zero-impact cases, using both a common absolute budget and a budget relative to the remaining pool. This would distinguish improved prioritization among consequential interventions from avoiding checks that can already be ruled out exactly.

  2. The incremental contribution and economic scope need clearer separation. Replacement already captures 29.04%, versus Decision's 30.24%, and this additional comparison is post-hoc. Under rank weighting, simple Allocation outperforms the transferred learned model. Moreover, the target is a six-session average absolute daily-return difference, which need not rank interventions like terminal wealth, risk-adjusted performance, or changes in strategy selection. These findings support decision-aware auditing, but more modest decision-distance baselines and sensitivity to an additional economic target would clarify the specific value of learning and the interpretation of recovered impact.

Questions for authors
  1. How much of Decision's advantage over Replacement survives allocation-equivalent deduplication and analytic zero-impact screening at the same absolute checking budget?
  2. Does prioritization remain effective for longer continuations or terminal-wealth differences, and how does it compare with a minimally parameterized volatility-aware decision-distance rule?
2 comments
AuthorsOpenReview ↗

Follow-up on economic equivalence and terminal-wealth results

Dear Reviewer drjT,

Thank you again for your concrete suggestions. We are following up on our response and the experiments added to address your questions.

After merging allocation-equivalent cases and removing analytic zeros, Decision improves over Replacement by 1.54 pp [0.80, 2.28] at the same 3,482 checks. For twenty-session terminal wealth, the gain is 2.04 pp [0.41, 3.66]. We also tested the volatility-aware decision-distance rule you suggested.

These results show that the learned gain survives stricter case construction and extends to terminal wealth.

We would appreciate your assessment of these results and welcome any follow-up questions.

Best regards,

The Authors

AuthorsOpenReview ↗

Follow-up on Additional Experimental Results

Dear Reviewer drjT,

Thank you again for your constructive feedback. We have provided additional results addressing allocation equivalence, zero-impact screening, and longer-horizon economic outcomes. We hope these findings help clarify the robustness and scope of our contribution.

We would greatly appreciate your thoughts on whether these results adequately address your concerns. We are happy to provide any further clarification.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.