- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates how weighting pairwise comparisons in learning-to-rank models can encourage unintended ranking behaviour, particularly learning return magnitude rather than direction. The authors propose SOGA, which identifies potentially harmful weighting schemes through a pre-training gradient diagnostic and adjusts them using constrained KL-divergence optimization. The approach is evaluated on Chinese and US equities and cryptocurrency, across several ranking objectives and architectures. The authors report improvements in cross-sectional RankIC, particularly against weighted objectives that otherwise perform poorly.
Critical strengths
The paper raises an interesting question about how pairwise weighting can affect what a ranking model ultimately learns. The proposed diagnostic is relatively simple, interpretable and applicable before training. The authors also provide a reasonable number of comparisons across ranking objectives and architectures, including uniform weighting and alternative correction methods. The experiments linking ranking deterioration to return magnitude are particularly useful.
Critical weaknesses
Novelty and positioning within the literature:
I am not entirely convinced by the novelty of the proposed approach. SOGA combines a gradient-based diagnostic with constrained adjustment of pairwise weights, but the paper does not sufficiently explain how this differs from existing approaches to ranking debiasing, importance weighting and constrained learning-to-rank. The literature review is also rather limited, particularly given the financial focus. Poh et al. (2021) is cited but barely discussed, despite its relevance to cross-sectional learning-to-rank and its more comprehensive financial evaluation and being one of the first to use LTR for cross-sectional strategies (see also follow up works by the same authors). Similarly, the relationship with Kwiatkowski and Chudziak (2025), from which several objectives are taken, deserves more attention. The authors should clarify what SOGA contributes beyond these existing methodologies and how the reported findings relate to previous results.
Empirical validation and performance reporting:
I find the empirical evidence insufficient given the exclusively financial application. The three Chinese equity datasets share the same approximately 2.5-year out-of-sample period, starting in June 2020 and therefore excluding the initial COVID-19 market crash, while cryptocurrency is evaluated only from July 2024 to December 2025, excluding the 2022 crypto collapse and FTX failure. These choices raise concerns about potential cherry-picking of evaluation periods, particularly given that the proposed mechanism concerns return magnitude and may behave differently during extreme market conditions. The absence of rolling or expanding walk-forward validation, despite the availability of much longer historical datasets, makes it difficult to rule out period-selection effects. Annual walk-forward testing would be a natural robustness exercise, particularly given the sensitivity of financial ranking to changing market regimes. The paper also does not systematically report RankIC standard deviations, ICIR or multiple-testing-adjusted p-values for improvements over uniform weighting, which are relatively modest compared with gains over poorly performing weighted objectives. Finally, performance is assessed primarily through average daily RankIC against four-day or ten-day forward returns. Although supplementary top-minus-bottom spreads are provided, there is no evaluation of risk-adjusted performance, turnover or transaction costs, as commonly reported in financial learning-to-rank research. Overall, the evidence suggests that SOGA can improve certain harmful weighting schemes, but does not convincingly establish robust improvements over competitive alternatives. The empirical evaluation is also confined entirely to financial return ranking, with no evidence that the proposed diagnostic generalizes to other ranking tasks or shortcut mechanisms. Given that SOGA is presented as a methodological contribution to learning-to-rank, this limits the broader significance of the findings.
Questions for authors
Novelty and Related Work: Could the authors clarify the methodological novelty of SOGA relative to existing ranking-debiasing and constrained reweighting approaches? In particular, how do the findings relate to Poh et al. (2021) (and their follow up papers) and Kwiatkowski and Chudziak (2025), which are cited but not discussed in sufficient depth? Generalisation: Could the authors demonstrate whether SOGA generalizes beyond financial return ranking, for example to information retrieval or recommendation systems? All current experiments concern financial assets and the same return-magnitude shortcut. Testing a different ranking problem and shortcut mechanism would help establish the broader applicability of the method. Walk-forward validation and potential period selection: Why was no rolling or expanding walk-forward validation conducted? The Chinese equity test periods begin after the initial COVID-19 crash, while the cryptocurrency test excludes the 2022 market collapse and FTX failure. These choices raise concerns about potential cherry-picking of evaluation periods. Given the available historical data, annual out-of-sample testing would seem feasible. How stable are the results across different market regimes and prediction horizons? Statistical Significance: Could the authors provide RankIC standard deviations, ICIR, paired confidence intervals and Holm-adjusted p-values for the main comparisons against uniform weighting? In particular, which improvements remain statistically significant after accounting for serial dependence and multiple testing? Financial Performance: Do the reported RankIC improvements translate into better risk-adjusted financial performance? A consistent long-short portfolio comparison reporting Sharpe ratio, realized volatility, maximum drawdown and turnover, ideally net of transaction costs, would help establish the economic significance of the results.
3 comments
R3. Walk-forward evaluation and period robustness
To address temporal robustness, we added an annual expanding-window evaluation across 25 equity market-years: CSI300 and CSI500 (2015–2022), CSI1000 (2017–2022), and US (2018–2020). Each test year uses earlier data only, separated by a (k+1)-day gap to avoid overlapping label horizons. The diagnostic, response tolerance and correction are recomputed from each training fold, so no test-period labels enter these steps.
SOGA improves weighted hinge v1 RankIC in every one of the 25 market-years, including periods of pronounced market disruption. Frozen LambdaRank is likewise repaired in every evaluated year. The pooled comparisons below quantify the improvement and place it alongside the uniform-weighting reference.
| Walk-forward test | Uniform RankIC | Hinge v1 RankIC | Hinge v1 + SOGA | SOGA vs uniform, widths (Holm p) |
|---|---|---|---|---|
| CSI300, 2015–2022 | 0.0797 | 0.0254 | 0.0880 | +2.82 (0.004) |
| CSI500, 2015–2022 | 0.0901 | 0.0100 | 0.0934 | +1.35 (0.18) |
| CSI1000, 2017–2022 | 0.0915 | −0.0077 | 0.0947 | +1.28 (0.18) |
| US, 2018–2020 | 0.0366 | −0.0413 | 0.0609 | +5.41 (< 10⁻⁶) |
| CSI panels pooled | — | — | — | +1.99 (0.004) |
The consistent improvements over uncorrected weighted objectives support robustness across market conditions and both label horizons (four days for CSI, ten days for US). Against uniform weighting, SOGA has positive RankIC differences on all four markets, with Holm-adjusted significance on CSI300, US and pooled CSI. We will report yearly and pooled results to make both aspects of the evidence clear.
R4. Statistical reliability and uniform-weighting comparisons
We have strengthened statistical reporting through paired daily analyses, within-period variability and seed-level variation. For every objective-market combination, we will report mean RankIC, daily standard deviation, ICIR and across-seed standard deviation, together with paired block-bootstrap confidence intervals, Newey–West tests and Holm adjustment within prespecified comparison families.
All 16 original single-split repairs relative to uncorrected weighted objectives are Holm-significant. Against uniform weighting, weighted hinge v1 + SOGA has higher mean single-split RankIC on all four markets, although these smaller differences are not Holm-significant. IC and winsorised IC exceed uniform significantly on all four single-split markets, and longer walk-forward RankIC comparisons are significant on CSI300, US and pooled CSI. These results substantiate the primary objective-repair claim while accurately distinguishing it from the comparison against a strong uniform baseline.
As a reference for variability, uniform weighting on CSI300 has mean daily RankIC 0.0632, daily standard deviation 0.141 and ICIR 0.45. Reporting this context alongside confidence intervals allows the magnitude and reliability of the improvements to be assessed directly.
R5. Cost-aware, risk-adjusted portfolio evaluation
We added a matched walk-forward long–short backtest for all objectives. Predictions made at the close of day t are traded at the close of t+1, with top-decile longs, bottom-decile shorts and k staggered holding sub-portfolios. We charge 10 basis points per side. Suspended A-share entries and those near the price limit (±9.5%) are excluded uniformly across methods.
Under this protocol, weighted hinge v1 + SOGA produces higher post-cost Sharpe ratios than uniform weighting on all four markets, with the CSI300 confidence interval including zero:
| Market | Uniform Sharpe | SOGA Sharpe | Difference [95% CI] |
|---|---|---|---|
| CSI300 | 2.49 | 2.79 | +0.30 [−0.01, +0.60] |
| CSI500 | 3.19 | 3.78 | +0.59 [+0.24, +0.94] |
| CSI1000 | 3.42 | 4.04 | +0.62 [+0.22, +1.03] |
| US | 0.58 | 1.05 | +0.47 [+0.06, +0.89] |
The additional risk metrics are also informative. Maximum drawdown improves on CSI300 (−15.6% versus −21.2%) and US (−25.0% versus −28.4%), and daily turnover is less than three percentage points above uniform weighting on each market. Much of the gain comes from the short leg, making short-sale availability and realised costs relevant to implementation. We will include these considerations while retaining the clear improvement under the reported evaluation protocol.
R6. Generalisation beyond financial ranking
SOGA’s diagnostic is defined by the response of a pair-weighted objective to a specified competing ordering, not by financial return magnitude alone. Its use in another ranking task requires a suitable candidate ordering, while the diagnostic and correction remain unchanged.
On KuaiRec recommendation ranking, we define magnitude as distance from the mean relevance label within a list. Three of four flagged tables exhibit harmful learned behaviour, and SOGA repairs these cases by 2.5–4.1 detection widths. This provides direct evidence that the same objective-level issue and correction can arise beyond financial ranking.
The competing ordering must oppose the task target: standard graded relevance in MSLR is aligned with that target and therefore is not itself a shortcut under our definition. We will clarify this criterion, distinguishing the general diagnostic formulation from the task-specific selection of a candidate shortcut.
R7. Cryptocurrency stress-period evaluation
We extended the cryptocurrency experiment to annual walk-forward test years 2022–2025, including the Terra/Luna and FTX disruptions. The equal-weight panel declined 75.7% in 2022. To handle disrupted trading histories consistently, returns spanning gaps longer than five days or documented unit changes are marked missing according to rules fixed before training.
Across these years, SOGA raises weighted hinge v1 pooled RankIC from −0.0144 to 0.0575, approximately 10 detection widths (Holm-significant), close to uniform weighting at 0.0585. The improvement therefore persists through severe stress periods and restores performance to approximately the uniform baseline. The central finding is robust repair of a harmful weighted objective, rather than an assertion of superiority to uniform weighting in every subperiod.
In 2022 alone the two are level, 0.3 detection widths apart, and in the Terra/Luna and FTX windows SOGA is slightly above uniform weighting (by 0.0002 and 0.0016 RankIC). The extended study provides an additional test of the proposed mechanism in a distinct, highly volatile market. We will include the cryptocurrency walk-forward and the stress-window results in the revision.
We thank the reviewer for the detailed feedback from a financial learning-to-rank perspective. We have expanded the evaluation with temporal validation, multiple-testing control, cost-aware portfolios and non-financial ranking tests. We also clarify the methodological distinction and related literature below.
R1. Methodological distinction from debiasing and reweighting
SOGA differs from existing approaches primarily in the source of bias it addresses. Label debiasing concerns supervision quality, importance weighting modifies the sampling or estimation distribution, and constrained ranking generally regulates the learned outputs. SOGA instead examines whether the pair-weight allocation within a specified objective rewards an unintended competing order before the learner is trained.
The diagnostic measures this preference from labels and pair weights at equal scores. Its correction removes the excess relative to a uniform reference while retaining the original objective’s intended emphasis. This is the substantive methodological contribution, supported in the submission by controlled relocation of identical weights, analysis of learned magnitude dependence and the derived objective-level correction. We will emphasise this distinction more prominently in the revised introduction and related work. The KL projection is the chosen optimisation mechanism rather than the basis of the novelty claim.
The distinction also has empirical consequences. Compared with a held-out-tuned penalty on the shortcut dependence of learned scores, SOGA improves frozen LambdaRank by 2.6–6.2 detection widths on all four markets, with Holm-adjusted significance. Correcting the objective before training therefore provides a different and more effective intervention than controlling the shortcut only in model predictions.
R2. Connection to financial learning-to-rank research
We appreciate the suggested references. They address complementary aspects of financial ranking, and we will clarify their relationship to the objective-level problem studied here.
Poh et al. (2021) compare ranking algorithms for cross-sectional investment and identify LambdaMART as a strong method on monthly NYSE data. Our work investigates a different stage of the pipeline: whether the pair weights within a ranking objective induce a competing order, regardless of the selected learner. To connect this analysis to established practice, we tested LightGBM’s built-in lambdarank. Its uncorrected objective underperforms uniform weighting by 4.0–7.8 widths across the four equity markets. Applying SOGA to the frozen LambdaRank table on the same learner recovers 3.6–7.5 widths, significant after Holm correction. This establishes the relevance of the diagnostic to a widely used ranking implementation.
Kwiatkowski and Chudziak (2025) compare ranking losses for stock selection, including formulations that emphasise high-ranked observations. Our weighted hinge v1 and v2 instantiate this type of emphasis, and the diagnostic flags both. In the walk-forward, SOGA raises weighted hinge v1 RankIC from 0.0254 to 0.0880 on CSI300 and from −0.0413 to 0.0609 on US. Their work establishes the practical importance of loss selection, and ours identifies a specific failure mechanism in pair-weight placement and provides a targeted repair. We will incorporate this relationship into the related-work discussion.