- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies the effect of uncertainty quantification in contextual optimization/predict and optimize setting. It particularly treats the setting where the prediction model relies on a distributional predictor and feeds it to the optimization problem to solve a downstream task. The paper claims that better UQ improves decisions only when a distributional feature is both imperfectly recovered and visible to the optimizer. In this setting, the paper claims that the uncertainty quantification has 2 sources : a statistical error and a decision error. They particularly look at this error by bounding the regret/excess risk of a projected distribution onto a structured family (location–scatter laws) by a finite sample estimation, a population misspecification and a decision sensitivity term. Experiments are run on retail inventory and industry portfolios and electricity procurement.
Critical strengths
The main strength of this paper is that it addresses an important problem in the DFL / contextual optimization community from a distributional point of view with respect to uncertainty quantification. It presents 2 theorems that deal with decision-risk stability and excess risk sensitivity.
Critical weaknesses
One of the major weaknesses is that the paper's results are very limited and can be summarized into a single main theorem that bound the excess risk of the methods. The main theorem presented only treats the projection of a learnt distribution into a structured family and states the results on that particular structure which limit the applicability of the theorem. The estimation term is only instantiated for location–scatter laws, and the misspecification term is never bounded for any family. The proof of the results rely on very classical results requiring the projection and standard W1 bounds for the location-scatter laws.
The results also rely on a triangular inequality which makes the clean separation between the terms loose. The decomposition in equation (10) is an upper bound and nothing proves its tightness. Additionally since the decision task is defined with respect to the learnt distribution, I would expect the two sources of uncertainty to interact which the paper does not explore. The latter also brings me to my next point about the paper not having proper positioning relative to prior work particularly with the predict and optimize (DFL) community. In the DFL community, most of the existing methods tend to rely on the point prediction algorithm and only a small number of methods rely on the distributional approach presented by the paper. Therefore a more systematic approach and a clearer presentation of the framework is needed to make the results more aligned with the DFL/predict and optimize literature.
Another weakness is related to empirical evaluation. As presented, the experiments are confusing and do not test the theory. The experiment claims to address the 2 sources of uncertainty presented in theorem 4.3 however they only present expected effects which are not representative of the presented theory. Most of the work in the experimental section focuses on the distinction between risk-neutral and risk-averse(CVaR) which concerns Proposition 4.4 rather than the main theorem.
Questions for authors
1- Beyond combining a projection step, standard W1 bounds for location–scatter laws, and the triangle inequality, what is technically new in Theorem 4.3? What are the main contributions beyond this theorem?
2- Can the misspecification term be bounded for at least one concrete family? Can the estimation term be extended beyond location–scatter laws?
3- Is the decomposition in Eq. (10) tight? Do the estimation and optimization errors interact, since the decision is made under the learned distribution? Also, the claim that UQ helps only when a feature is imperfectly recovered and visible to the optimizer is a necessity claim. What result supports it, given that Theorem 4.3 is only an upper bound?
4- How does this work relate to DFL and predict-then-optimize methods? Do decision-focused losses control δ_f? When should a distributional predictor outperform a decision-focused point predictor?
5- Which experiment tests which term of Theorem 4.3? Most experiments compare risk-neutral and CVaR objectives, which relates to Proposition 4.4 rather than the main theorem.
6- What is the purpose of Sections 4.3 and 4.4? Why consider moment-based decisions, and why introduce CVaR?