- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper trains two heads together on nine equity indices: a regression head for the 4-day-ahead return, and a 61-class state classifier with an explicit right-tail class (return > 3%). A one-way squared hinge pushes the regression output above the tail threshold whenever the classifier predicts the tail class. The inputs are five tokens, obtained by 1-D K-means on the RMS magnitudes of overlapping OHLC windows. The evaluation covers two test periods and three seeds. It reports MSE/MAE and tail AUPRC/AUROC, with the regression output as the score, plus loss ablations and a backtest.
Critical strengths
Table 3 is a clean comparison: the same backbone with ten objectives, including a ground-truth hinge control. The proposed term improves the regression output's AUPRC over MSE+CE in both periods (0.111 to 0.146, 0.084 to 0.133). The protocol avoids leakage: the splits are chronological, and K-means and scaling are fit on training data only.
Critical weaknesses
My main concern is that the authors' own released run does not show the central claim (threshold-consistent coupling), and the other gains are marginal.
-
The coupling does not do what the paper claims. I looked at the saved output of the released notebook (2021-2025, seed 1, default settings, with one extra dropout draw). The classifier predicts the tail class for 350 test samples, but the regression output exceeds 1.03 for only 3 of them. During training, the crossing rate stays at 1-3%. So the hinge acts as an upward bias on flagged samples, not as a consistency constraint. In the same run, the classifier's tail probability is a better tail score (AUPRC 0.154, AUROC 0.636) than the regression output (0.146, 0.621). The paper evaluates only the regression output, so it never shows that the coupling adds anything over using the classifier directly.
-
The gains are small and the comparisons are uneven. The Random Walk has lower MSE in both periods (Table 2). TCDL's MAE advantage over it is 0.15% and 0.03%, and the paper does not test it, so the claim of a clear point-forecast advantage (L261-262) is not supported. Tail AUROC is about 0.59, and Autoformer has higher AUPRC in both periods. The baselines predict the close price with MAE loss and identity scaling, while TCDL gets normalised inputs and a standardised return target. There is also no volatility baseline, which is the obvious reference for 3% moves.
Questions for authors
-
Please report tail AUPRC/AUROC with the classifier probability as the score, for MSE+CE and TCDL. Please also report the fraction of argmax-tail samples with r̂ > 1.03, for all seeds and both periods.
-
The notebook's seed-1 tail metrics (0.146 / 0.621) are outside what Table 2's three-seed mean ± SD allows (0.1326 ± 0.0062 and 0.5896 ± 0.0121). Which code produced Table 2, and how large is the seed variance over more seeds?
-
Please add per-index Diebold-Mariano tests against the Random Walk and a volatility baseline (realised volatility or GARCH-implied P(r > 1.03)). Does the score also rank left-tail events?
-
How were the defaults (0.5, 0.1, 60) chosen? They give the best test AUPRC in all six groups of Table 6.
-
For the backtest, please compare against random-timing overlays with the same activation rate and against baseline signals, with a significance test.
Related work that may be of interest: Ding et al. (KDD 2019) and NEC+ (Li et al., AAAI 2023).
6 comments
Response to Reviewer L4yS (2/2)
Q1. Classifier-probability metrics and the fraction of argmax-tail samples with r̂ > 1.03
Reviewer: report classifier-probability AUPRC/AUROC for MSE + CE and TCDL, and the fraction of argmax-tail samples with r̂ > 1.03 for all seeds and both periods.
Both requests rest on the two misreadings described above: the classifier cannot provide a point forecast, so its tail metrics fall outside the scope of the paper, and the crossing loss is a training-time penalty, so the test-time crossing fraction does not measure the mechanism (W1). Nevertheless, the requested fraction is higher with the crossing loss even at test time, where the loss is not applied:
| Period | Model | Seed 1 | Seed 42 | Seed 1337 | Mean |
|---|---|---|---|---|---|
| 2015–2019 | MSE + CE | 5.4% | 6.9% | 4.7% | 5.7% |
| 2015–2019 | TCDL | 9.0% | 5.2% | 8.3% | 7.5% |
| 2021–2025 | MSE + CE | 0.5% | 0.0% | 0.0% | 0.2% |
| 2021–2025 | TCDL | 0.6% | 0.4% | 2.2% | 1.1% |
The fraction remains small, as expected for a soft training-time penalty with weight 0.1 under a dominant MSE term; what the paper claims is the improved ranking of the regression output (Table 3).
Q2. Which code produced Table 2, and how large is the seed variance?
Reviewer: the notebook's seed-1 metrics (0.146/0.621) fall outside what Table 2's mean ± SD allows.
The released notebook is the code that produced Table 2, including the extra dropout call noted by the reviewer. We have uploaded the training log (backtest_2021_seed1.log) and the per-sample test predictions (test_predictions.csv) of the Table 2 run for 2021–2025, seed 1. They show the same model with 178,558 parameters and give AUPRC/AUROC of 0.1370/0.6036; together with seeds 42 (0.1255/0.5825) and 1337 (0.1353/0.5828), this yields exactly 0.1326 ± 0.0062 and 0.5896 ± 0.0121. The outputs saved in the released notebook come from a Google Colab run, whereas Table 2 was run on a local CPU (macOS, Python 3.9, PyTorch 2.8.0). PyTorch does not guarantee identical results across platforms even with fixed seeds, which is why we report three-seed statistics; within a fixed environment, the seed variance is the SD reported in Table 2 (0.0062 for AUPRC in 2021–2025). The Colab run is better than ours, so Table 2 does not reflect a selection of favorable runs.
Q3. Diebold–Mariano tests, a volatility baseline and left-tail ranking
Reviewer: per-index Diebold–Mariano tests against Random Walk, a volatility baseline, and whether the score also ranks left-tail events.
The paper does not claim to beat Random Walk in point error; it claims to match it while providing tail ranking, which Random Walk cannot provide: its AUROC is 0.5 and its AUPRC equals the tail rate (6.7% in 2021–2025) (W2), so a test of equal accuracy would not change the conclusion. A GARCH-implied gives no point forecast and rises with predicted volatility, so it scores large falls as highly as large rises. Left-tail ranking is outside the design: the task targets right-tail opportunities, with a single explicit right-tail state and a one-sided crossing loss.
Q4. How were the defaults (0.5, 0.1, 60) chosen?
Reviewer: the defaults give the best test AUPRC in all six groups of Table 6.
They are a priori round values (the classification term at half the weight of the regression term, a small weight for the auxiliary crossing term, and an intermediate vocabulary size), fixed before the sensitivity analysis and shared by both periods and all nine markets; Table 6 is a post-hoc, one-at-a-time analysis around them. The defaults lead only in AUPRC: in 2021–2025, other values give lower MSE (0.000484 vs. 0.000488), lower MAE (0.015774 vs. 0.015831) and higher AUROC (0.5953 vs. 0.5896), which a configuration tuned on test data would not do. The AUPRC lead is also often within seed variation (e.g., λcls = 0.10 gives 0.1322 ± 0.0095 vs. 0.1326 ± 0.0062).
Q5. Random-timing overlays, baseline signals and significance
Reviewer: compare the backtest against random-timing overlays with the same activation rate and against baseline signals.
The backtest illustrates the economic use of the regression output, and Section 5.5 concludes only that the scores "may provide useful information for dynamically adjusting exposure"; models are compared on the forecasting and tail metrics in Table 2. A random-timing overlay cannot raise the Sharpe ratio: applying the 0.5× overlay in a fraction of periods independently of returns gives at most times the Buy-and-Hold Sharpe ratio (about 0.99 at ), before transaction and borrowing costs reduce it further. The TCDL overlay instead raises the average Sharpe ratio in all six settings of Table 4 (0.4333 → 0.4401–0.4603; 0.5307 → 0.5418–0.5783; +1.6% to +9.0%) despite costs, so the gain reflects signal timing rather than extra exposure.
Thanks for the reply and for uploading the log and predictions.
I recomputed the metrics from test_predictions.csv and get 0.1370 / 0.6036, which is consistent with Table 2, so I withdraw that concern. I also put the crossing-rate point too strongly, given that Section 4 describes a soft penalty, though I would still drop "logically aligned" from the abstract and conclusion.
My main remaining concern is the volatility baseline (my Q3), so I computed it from your files: trailing 20-day realised volatility at the forecast origin, no fitting (the date column in the CSV is the target date, so the origin is four trading days earlier). On the same 11,192 rows it gets AUROC 0.723 vs 0.604 for TCDL, and AUPRC 0.166 vs 0.137. Its AUROC is also higher in each of the nine markets (mean 0.693 vs 0.612). You're right that volatility isn't direction-specific: it ranks falls of the same size (r < 1/1.03) above chance too, at 0.641. But so does TCDL's output (0.562), and among the 1,520 moves beyond 3% either way, volatility separates rises from falls about as well (0.589 vs 0.546). For 2015-2019, I rebuilt the test set from the same price files (11,213 rows vs your 11,180) and get AUROC 0.749, vs 0.594 for TCDL in Table 2.
A simple non-learned score ranks the right-tail events better than the model, and I think the paper needs that comparison. I'm raising my score to 2 since the reproducibility issue is resolved, but I still lean towards rejection because of this.
Response to Reviewer L4yS (1/2)
We mean no offense and simply wish to state the following facts honestly. After completing this paper, we gave it to large language models (e.g., ChatGPT and Claude) for a pre-submission review. However much we emphasized these points, the models consistently made the same two misreadings:
We found both misreadings, in full, in this review. Reading it late at night, we admit that our blood pressure went up.
W1. The coupling does not do what the paper claims
(1) Training-time penalty, not a test-time constraint. Section 4 defines the crossing term as "only a training-time penalty" that transfers state information "without forcing threshold-consistent predictions toward the decision boundary." It shapes the regression head during training; the paper never reports or claims a test-time crossing rate. What we claim and evaluate at test time is whether the regression output ranks right-tail events higher (AUPRC/AUROC), and these rank-based metrics do not require predictions to exceed 1.03. Because MSE remains the dominant term, predictions stay shrunk toward the mean, so few cross 1.03 even when their ranking improves. Counting test-time crossings therefore does not evaluate the mechanism.
(2) A low crossing rate means the constraint is active. The loss acts precisely on predicted right-tail samples that have not crossed the threshold. A crossing rate of 1–3% means the penalty acts on about 97% of them in every epoch, and their mean squared shortfall decreases from 0.603 to 0.433 (about 28%) during training (see our response to Reviewer xxSR, Q1).
(3) What the coupling adds. With the same backbone, classification head and CE loss, adding the crossing loss raises the AUPRC/AUROC of the regression output from 0.1106/0.4706 to 0.1464/0.5935 (2015–2019) and from 0.0842/0.4596 to 0.1326/0.5896 (2021–2025), with nearly unchanged MSE; the same hinge applied to ground-truth tail samples yields smaller gains (0.1252/0.5532 and 0.1124/0.5848; Table 3), so the benefit comes from conditioning on the model's own predicted state.
W2. Small gains and uneven comparisons
Each comparison must be read against both goals of the paper: low point error and informative tail ranking.
(1) Random Walk cannot rank or identify tail events. It predicts r=1 for every sample, so its AUROC is 0.5 and its AUPRC equals the tail rate (hence N/A in Table 2); its low MSE reflects the concentration of four-day return ratios near 1 (Section 5.1). Pure point-error minimization drifts toward this flat forecast, the failure mode motivating the paper (Section 3). TCDL matches its point accuracy (MSE within 0.000005 and 0.000003, lowest MAE in both periods) while also ranking tail events. Significance tests are therefore run against the learned baselines, the only models that can serve both purposes: the MSE and MAE gains are significant against all eight in 2015–2019 and six of eight in 2021–2025.
(2) Higher AUPRC comes at a much larger point error. Autoformer's higher AUPRC (0.1503, 0.1684) comes with about three times TCDL's MSE (0.001616 vs. 0.000522; 0.001303 vs. 0.000488), and TimeXer's (0.1404 in 2021–2025) with larger MSE and MAE. Such models favor one goal at the expense of the other (Section 5.1).
(3) Same target, same evaluation. All models predict the four-day-ahead close, and all tail metrics use the regression output. The objective differs because it is the factor under study; with identical inputs, normalization and backbone (Table 3), tail AUPRC rises from 0.0923 (MSE only, i.e. the same model trained as a standard regressor) to 0.1464 and from 0.0864 to 0.1326.
(4) Volatility models address a different task. They forecast the scale of returns, not the return itself, and provide neither a point forecast nor a direction-specific ranking.