- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors compare stock-ranking models over twelve quarterly walk-forward folds and ten seeds. They use selection-adjusted tests and fixed-hyperparameter graph interventions. The tuned correlation-GAT has lower IC than the MLP. After selection adjustment, no model is established as superior to LightGBM.
Critical strengths
-
Purging, multiple seeds, and multiplicity corrections make the study more informative than a single tuned comparison. The 90-trial sensitivity analysis, BY correction, and lag-21 check strengthen the result for the tested correlation-GAT setting.
-
The negative findings have value. The paper reports GraphSAGE collapse, gains concentrated in a few quarters, and a news-edge IC improvement that does not carry over to Sharpe. These results help explain the limits of adding graph information in this setting.
Critical weaknesses
-
Universe B is described as leak-free, but Appendix C.1, p.18, lines 946–948, says a diagnostic on 63 days within the confirmatory test span informed adding volume features. Test observations therefore contributed to feature selection. Freezing the features before a later run does not remove this dependence. Shared features may affect models differently; the resulting bias is not quantified.
-
Section 3.2 estimates Sharpe from about three returns per fold-seed cell and averages the ratios. These volatility estimates are unstable, and their average is not the Sharpe of a chronological strategy. Bootstrapping twelve fold differences does not address that distinction. The cost conclusions in the abstract and Section 5.3 therefore need a clearer basis. The reported 16.3% composition mismatch in the non-PIT panel also limits interpretation of portfolio and graph results.
Questions for authors
-
Which feature and model decisions preceded inspection of any test-window results? Is there an existing price-only result or untouched segment supporting the correlation-GAT versus MLP comparison?
-
Do the cost conclusions hold for stitched chronological portfolio returns? Please use the existing outputs and state how seeds and portfolio costs are handled.
2 comments
Response to Reviewer 9tq9 (2/2)
Part 2 — W2 / Q2: Sharpe, stitched portfolios, and composition
We agree. A Sharpe ratio from about three returns says little, an average of such ratios is not the Sharpe ratio of any strategy, and bootstrapping fold differences does not fix that. The confirmatory run saved per-cell metrics and daily IC but no predictions or portfolio returns, so we re-ran the confirmatory cells with prediction logging, using the frozen hyperparameters and the same ten seeds. L0 and L1, re-run on the confirmatory hardware, reproduce the confirmatory daily IC bit for bit in all 480 cells. L2, L3 and L5 were re-run on an A100 (confirmatory: T4); their pooled IC moves by at most 0.0004, except C L5 (−0.0017), which we use descriptively only (Appendix E.2).
How the portfolio is built (§3.2, Appendix E.3). One 21-trading-day rebalance grid starting 2023-01-03 gives 35 complete holding periods that end 2025-12-08. At each rebalance we go long the top 50 and short the bottom 50 of the 501 names (equal weight, dollar neutral), using the predictions of the fold whose test window contains that date. Positions carry across fold boundaries, eligibility uses entry-date information only, and positions are marked daily for drawdowns. Seeds: each seed gives one portfolio. Paired ΔSharpe is the seed mean of the per-seed Sharpe differences. The 95% interval comes from a stationary bootstrap over periods (mean block 3, 5000 draws) that resamples the same periods for both arms and all seeds, so the interval is conditional on the ten fitted seeds. Costs: net return = gross − τ·c/10⁴, charged at every rebalance, where τ is the L1 change in the two books' equal target weights (4 at the first entry). τ is a turnover proxy: it ignores weight drift between rebalances. There are no financing or borrow costs.
Results at 10 bps (Table 4).
| Pair | Gross ΔIC | Net ΔSharpe [95% CI] | Range over 21 phases (# positive) | Submission |
|---|---|---|---|---|
| C L1−L0 | +0.0148 | +0.29 [−0.51, +0.88] | −0.45 to +0.44 (15) | +1.17 |
| C L2−L1 | −0.0119 | −0.02 [−0.20, +0.17] | −0.35 to +0.02 (1) | −0.72 |
| C L3−L2 | −0.0123 | −0.41 [−0.74, −0.10] | −0.68 to −0.23 (0) | +0.08 |
| C L5−L3 | +0.0275 | +0.44 [+0.16, +0.74] | +0.22 to +1.09 (21) | +0.88 |
| B L1−L0 | +0.0143 | +0.46 [−0.38, +1.26] | −0.13 to +0.62 (19) | +0.43 |
| B L2−L1 | −0.0133 | −0.41 [−0.65, −0.24] | −0.42 to −0.02 (0) | −0.14 |
The primary estimate uses 35 holding periods; each phase-sweep portfolio uses 34, so that all 21 rebalance offsets can be compared. The two Universe-C contrasts that the submission said "survive 10 bps" now have intervals that include zero. Applied to the same re-executed cells, the submission's averaging gives +1.17 and −0.64 (submission: +1.17 and −0.72), so the change comes mainly from the estimator, with some re-execution drift. We withdraw the statement that these contrasts survive costs. We also withdraw the submission's statements that the tuned news-edge IC loss is "near zero in net Sharpe" (abstract) and that the news-edge rung "matches" the correlation-GAT in net Sharpe (§5.3, +0.08): with the stitched portfolio, the news-edge rung is lower (−0.41 [−0.74, −0.10]). For Universe B, the submission reported a cell-averaged L2−L1 of −0.14 with an interval containing zero; the stitched estimate is −0.41 [−0.65, −0.24] and is negative at every rebalance phase. Timing matters: C L1−L0 changes sign across phases. Bonferroni adjustment over the six contrasts keeps B L2−L1 and C L5−L3 away from zero but not C L3−L2. A 36th period, the historical-membership restriction, other block lengths, and a tie-robust treatment of LightGBM's tied predictions change no sign and no interval's relation to zero (Appendix E.3). Net Sharpe remains descriptive.
Composition (16.3%). §4 now reports the survivorship audit. Removing, on each date, the snapshot names that had not yet joined the index (50 at the first test date) lowers every arm's pooled IC by 0.004–0.012. Most paired contrasts change little. Universe-B L2−L1 weakens from −0.0130 to −0.0086 [−0.0176, +0.0005] (unadjusted HLN p between 0.02 and 0.16, depending on the HAC kernel and bandwidth). Universe-C L2−L1 is −0.0113 [−0.0177, −0.0049]. C L3−L2, whose full-snapshot interval only just excludes zero, moves from −0.0124 to −0.0101 [−0.0221, +0.0009] (Appendix E.4). The restriction cannot restore the 87 departed firms, which are also missing as graph neighbours, so survivorship remains a limitation whose effect on graph-versus-non-graph comparisons we cannot sign (L8).
Response to Reviewer 9tq9 (1/2)
Part 1 — W1 / Q1: Universe B and the decisions that preceded test-window results
Thank you for reading Appendix C.1 this closely. You are right: the main text described Universe B more strongly than C.1 supported. While re-checking the project log for this response, we also found two exposures that the submission did not state. The revision reports both:
What changed. We removed every "leak-free". §4 now states that B's basis (ten standard price-volume features) was fixed in April 2026 and not changed afterwards, and that this basis is not free of evaluation-window information. Specifically, the 63-day diagnostic (2024 Q2) scored six of these features and informed the addition of the two volume features, and the later ranking scored all ten. We designated B as primary after the Universe-C leakage audit and after the confirmatory run. Limitation L1 now calls B "our closest comparison outside that ranking" and says that it does not bound the selection-induced inflation in Universe C. §5.2 no longer argues that selection cannot affect the negative graph contrast.
Q1: which decisions came before test-window results. Appendix E.1 (Table 22) dates every step against the test quarters whose realized labels it used. In short:
Is any segment untouched? No. The 2023 quarters (folds 0–3) were never used for validation or testing before the freeze, but the untuned pilots evaluated them before the hyperparameters were frozen and before Family-2 was added. We therefore present no segment as a holdout for the final protocol, and we agree that this design cannot quantify the selection bias you describe.
Price-only check. We added a targeted sensitivity, Universe B8, which deletes B's two volume features (dollar volume and the 5-day price-volume correlation), so the remaining eight features use prices only. With the frozen Universe-B hyperparameters, L2−L1 is −0.0181 [−0.0310, −0.0065] (11/12 folds negative), so the negative contrast remains after this deletion. The change in this contrast relative to B is −0.0051 [−0.0164, +0.0062], so we cannot say whether the magnitude changed. B8 still contains six features that the diagnostic scored, so this subset does not remove the dependence on evaluation-informed selection. The same deletion weakens the MLP−LightGBM contrast from +0.0143 to +0.0021, mainly because LightGBM's pooled IC rises (0.0228 → 0.0358). This fits the paper's reading of MLP > LightGBM as suggestive only (Appendix E.6, Limitation L1).