Backtest Mirage: Decision-Aware Auditing of Financial Machine Learning

Small errors in a model's scores can change the economic conclusion of a financial backtest. A small change near the selection cutoff can replace a stock, while a much larger change elsewhere leaves the portfolio unchanged. We introduce decision-aware economic auditing: rank candidate interventions by the investment decisions they change, then measure economic impact through paired portfolio continuations. Across all 24 future-input conditions, exposure near the cutoff produces smaller score changes but larger economic deviations than exposure farther away. On CSI 300 and CSI 500 with Ridge and LightGBM, our learned Decision priority captures 34% more impact than a learned score-only baseline at the same 10% checking budget in a frozen 2025 test. In a secondary rank-weighted evaluation, exploratory Allocation captures impact from weight changes among the same stocks. Decision information thus directs a fixed checking budget toward errors with larger effects on the economic evidence from backtests. Code is available at https://anonymous.4open.science/r/Backtest-Mirage-0220.