- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a new idea that we could use pairwise outcome order to fulfill the causal matching work instead of relying on outcome magnitudes. Its assumption of a Gumbel location model successfully builds the bridge between order and magnitude that makes the ATT estimation mathematically plausible. Besides, the author also tries to define the bound by linking score-learning error, future donor availability, and donor reuse to ATT MSE, making the error of ATT estimation explainable by errors from RankNet's prediction. The paper validates the method using several synthetic settings, a sample-size study, and a semi-synthetic LaLonde experiment. RankNet is compared with different matching methods such as full-outcome regression methods, BART, and causal forest. The experiments try to cover different outcome structures and noise settings and also report cases where RankNet does not outperform the alternatives.
Critical strengths
- The author brings a new idea that pairwise outcome order could replace the raw outcome magnitudes for control selection in a causal problem. This helps to understand how much outcome information is actually needed for learning useful matches from a different angle
- The math proof is concise and straightforward, as it builds the connection between order and prognostic distance under Gumble assumption.
Critical weaknesses
-
The practical motivation of the proposed method could not be clearly detected. The method is developed and tested under lots of assumptions. And I don't know why or when I need to choose the method over other methods when the historical outcome is available. Especially, the experiments show that full-outcome learners can perform substantially better than RankNet.
-
The empirical evaluation relies heavily on ATT MSE. Instead of evaluating it like a predictive model, I think more metrics for evaluating causal matching methods should be considered. We also care about the matching quality instead of the final ATT estimation.
Questions for authors
-
In what practical setting should we use pairwise outcome order instead of the original historical outcome values? If the numerical outcomes are already available, especially given that full-outcome methods such as Huber regression perform better in some experiments, what is the main benefit of discarding the outcome magnitudes?
-
What is the main new contribution of Theorem 1 compared with existing matching theory? The connection between score error and matching error could also apply to a prognostic score learned directly from numerical outcomes, so how would you attribute the contribution to pairwise order comparison?
-
How do we know that RankNet's improvement over MALTS comes from using pairwise-order information but not from RankNet trained with a more flexible neural network? This is particularly important for the folded prognosis experiment, where the chosen setting appears to favor RankNet's nonlinear representation.
-
Beyond overall ATT MSE, can the authors provide additional evidence that RankNet actually produces good causal matches? For example, metrics such as ATT bias and variance, matching quality, donor reuse, and post-matching balance could help validate the method and explain why it succeeds or fails in different settings.
-
Related work mentioned Li and Fu (2017) as its closest ordinal matching precedent, but does not include it in the experiments. Is possible, it would also be great to compare RankNet with this method in experiments.
Thank you for the constructive review. We have made substantial changes in the revision. Below is our point-by-point response. Our general response (the Official Comment by the authors) addresses novelty and clarifies when rank methods help.
W1 / Q1. Practical motivation: why discard magnitudes when outcome is available? In short, we discard magnitudes only when they are on the wrong scale so regression on them then learns the wrong conditional mean, while ranks still encode the outcome distribution (Figure 1, Section 3.1). In an applied example, a device or assay changes, as with the NHANES serum-folate assay, historical outcomes may not share the future scale. When values are reliable, regression should be used, and the paper shows this (Section 4.4; Section 4.2 in the original submission).
Q2. Contribution of Theorem 1. We address your second point first. Theorems 1 and 2 now share one framework for comparing learners: the affine calibration error a(g^), a quantity that works for any learner, so RankNet and regression are judged on the same scale. That the bound also applies to regression is therefore deliberate.
On your first point, what is new relative to existing matching theory is when RankNet wins. Under a measurement transformation, ranks still encode the outcome distribution (Proposition 1), while outcome regression learns the transformed, and therefore biased, conditional mean. In the toy example, a=0 for RankNet versus a≈0.122 for regression (Section 3.3). Theorem 2 characterizes the conditions under which RankNet is better in general, and the simulations confirm this (Section 4.3).
Q3. Ordinal information vs network flexibility. Note that MALTS is an outcome-based method, and the Huber MLP is a more flexible outcome-based method with the same two-layer network as RankNet. The gain over Huber is therefore a gain over outcome-based learning at equal flexibility: under distortion, Huber degrades while RankNet does not. We also added a normal-rank MLP with RankNet’s architecture, bounded head, initialization and assignment rule, trained on rank-derived targets. It is equally invariant and has similar MSE (paired difference −0.126 (0.284), Appendix B.8). The benefit thus comes from ordinal supervision, not from network flexibility or the pairwise loss.
W2 / Q4. Matching quality beyond ATT MSE. Good point. We agree that ATT MSE alone hides how the matches are formed. We now report the individual matching discrepancy Dq, a closeness measure, together with Bq, a bias measure (Section 4.2, Figure 2). Bq and Dq are related: since Bq2≤Dq, a small Dq is the stronger result, because a small bias can come from cancellation, while a small discrepancy cannot. The results show that RankNet is better in MSE, bias and matching discrepancy, though the bias advantage holds at every transformation level, while the MSE and matching-discrepancy advantages appear under a moderate transformation of the outcome.
Q5. Li and Fu. We added it to the benchmark (Section 4.4, Figure 3; Appendix B.11). It is an outcome-based method, so we fit it on both historical arms and then match the future cohort. It does not perform as well: it is worse than RankNet in the linear, folded and KS designs and similar in the nonlinear design.