- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The study investigates the differences in agreement between the two methodologies of Least-Squares Estimation and Optimization (ETO) and Decision-Focused Estimation; and presents an approach to combine the two methods using a principle-based interpolation method. An Entropically Smoothed SPO+ loss is presented which maintains Fisher Consistency under symmetric noise, and can be differentiated with respect to the estimated parameters. This allows for both Sandwich Covariance Estimation and a Quadratic Test Statistic to detect when there is disagreement between the estimators. The same statistic is utilized to select a Closed Form Interpolation Weight. Asymptotic Calibration, Oracle Results, Dominance Results, and SURE Safe Last Layer Variant Variants (JS-W and Cross-Fitted) are all provided in the paper. Experimental evaluations have been conducted on a variety of data types such as Synthetic Data, Combinatorial Optimization Data, Electricity Market Data, and Neural Prediction Data. These evaluations demonstrate that tests are generally well-calibrated and interpolation is beneficial.
Critical strengths
The single most significant methodological advance in this study was using the identical quadratic measure to quantify the difference in estimator performance and to estimate the unknown variance (or discrepancy) in the local path risk. By smoothing both components of SPO+ it is possible to maintain the same symmetric consistency properties associated with SPO+, yet enable estimation of the joint covariance of these components. The analysis of the data-dependent nature of the shrinkage weight using a Stein correction, as opposed to simply replacing the data dependent weight with a random weight within a fixed-weight risk formula, is another important feature.
Furthermore, the differences in the SURE dominance criterion and the wider range of conditions under which the safer constants apply, adds to the informational content of the statistical model, as opposed to a purely ad-hoc interpolation heuristic. Once the two fits and the covariance terms have been determined, evaluating the resulting weight is computationally inexpensive.
Critical weaknesses
The question in the paper is whether decision-focused training will alter the decisions made by the trained model. The paper repeatedly refers to the rejection of the hypothesis (that decision-focused training will not alter the decisions) as an indication that the model or last layer should be moved.
However, the formal null hypothesis for this study is that there are equal population parameters. The authors acknowledge, as stated in Section 2 and Remark S8-H, that equal population parameters do not necessarily mean that the decisions made by the two models will be the same; positive scaling may result in different parameter values but no change in decision, and asymmetrical error distributions may lead to the rejection of the null hypothesis while ETO is already optimal for making decisions.
Moreover, Proposition S7-d supports calibration with a correctly specified conditional mean and symmetrically distributed conditional errors. Calibration does not support the null hypothesis of equal population parameters.
Additionally, the authors conclude their discussion by stating that they have no level guarantees for adequate models with asymmetrical errors or an inaccurately specified mean. As such, the reported p-value has limited applicability in the larger context that motivated the paper.
Questions for authors
What is the exact conclusion that practitioners will be able to make based upon rejection (of their model) in addition to disagreements regarding the ETO/Smoothed-SPO+ population target?
Specifically, what additional assumptions are required to allow one to conclude that the decision-making process needs to be altered or that decision-based training may help improve it? What is the distinction between a formal parameter-equality null hypothesis, a formal equality of hard decisions null hypothesis, and the fact there is no way to achieve a reduction in regret?
In addition, Proposition S7-d states that there is no general level guarantee for certain distributions which satisfy parameter equality. Does this mean that the authors believe they have developed a calibrated test within the correctly specified symmetric model class and not a calibrated test across the entire decision-adequacy null?