- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes a robust optimization framework that applies an outer risk functional to the expected loss under randomly reweighted empirical measures. It establishes an equivalent DRO representation, studies a Dirichlet–Gaussian family, and connects the framework to covariance shrinkage. For CVaR as the outer risk functional, the authors develop a stochastic subgradient algorithm for Dirichlet weights.
Critical strengths
The paper provides a unified framework with several interesting theoretical properties. In particular, the Dirichlet–Gaussian construction and the discussion in Section 3 (including Figure 1) help explain how the choice of weight distribution affects the resulting risk.
Critical weaknesses
- The paper is difficult to follow due to dense notation and insufficient explanation of the motivation for introducing a distribution over empirical weights and an outer risk functional. Theorem 1 appears to follow directly from the standard risk-envelope representation and linearity of expectation.
- Although Appendix E discusses standard convergence conditions, no convergence rate or finite-iteration guarantee is provided for Algorithm 1.
- Experiments are limited to a single small dataset, with marginal and metric-dependent improvements over -SVM. The empirical evidence is insufficient to establish the practical advantages of the proposed framework.
Questions for authors
- What are the concrete advantages of specifying an outer risk functional and a distribution over empirical weights rather than directly defining a DRO ambiguity set?
- While negative weights increase variability in convex order, could the authors provide empirical evidence or an intuitive explanation of how this might improve robustness under distribution shifts?
- Algorithm 1 relies on Monte Carlo sampling, and the proposed approach introduces an additional hyperparameter selected through cross-validation. How does the overall computational cost compare with standard DRO/CVaR methods? A discussion of computational complexity would be helpful.
- Given the marginal improvements on a single dataset, can the authors provide stronger empirical evidence to demonstrate the practical advantages of the proposed framework? Additionally, can the authors clarify the respective contributions of Dirichlet and Gaussian weights through ablation studies?
2 comments
Q2.
While negative weights increase variability in convex order, could the authors provide empirical evidence or an intuitive explanation of how this might improve robustness under distribution shifts?
For fixed sample losses, is linear in . Gaussian perturbations to weights therefore give a Gaussian centered at that observation’s loss. Intuitively, This adds Gaussian noise to the sample losses (not the observations ), letting the risk functional consider losses beyond the observed range. See Figure 1 in the paper for visual illustration.
Specifically, For , Section 3.3, Eq. (11), gives
\sqrt{-\frac{\tau}{1+\tau}}\\,\operatorname{sd}(X)G, \qquad G\sim\mathcal N(0,1),\qquad G\perp X. \tag{11}$$ Here $X$ is a uniformly sampled loss from $\\{X(\xi_i)\\}_{i=1}^n$. For empirical evidence, the additional experiment on QSAR selects Gaussian weights in all folds at $\nu=0.50$ and $0.40$. DRO can also improve out-of-sample performance without a change in the data-generating distribution by addressing sampling uncertainty \[2\]. Our experiments concern this setting. Controlled distribution-shift experiments remain important future work. ## Q3. > Algorithm 1 relies on Monte Carlo sampling, and the proposed approach introduces an additional hyperparameter selected through cross-validation. How does the overall computational cost compare with standard DRO/CVaR methods? A discussion of computational complexity would be helpful. We compare the arithmetic cost of one joint objective/subgradient evaluation, using $n$ observations and dense linear losses with $d$ features. For empirical CVaR \[3\] and the standard dual formulations of KL- and $\chi^2$-DRO \[4, Eq. (5)\], evaluating the losses and forming the weighted model gradient costs $O(nd)$. The remaining scalar terms cost $O(n)$, giving $O(nd+n)$ per full-data evaluation. For Algorithm 1, evaluating the sample losses once costs $O(nd)$. Drawing $M$ Dirichlet weight vectors and accumulating $\widehat q$ costs $O(Mn)$. The model gradient $\sum_i\widehat q_i\nabla_\theta X_\theta(\xi_i)$ is then computed once in $O(nd)$, giving $\mathbf{O(nd+Mn)}$ per evaluation. The $M$ draws therefore share one computation of the model gradient. For Gaussian weights, the analytic Gaussian mixture objective and gradient in Appendix C require $O(n)$ scalar operations, giving $O(nd+n)$ without Monte Carlo sampling. Thus the Gaussian branch has the same per-evaluation order as these classical reformulations; the Dirichlet branch adds $O(Mn)$ sampling and aggregation work. Selecting $\tau$ from $K$ candidates with $J$ inner folds requires $JK$ fits plus a final refit at each fixed $\nu$. ## Q4. > Given the marginal improvements on a single dataset, can the authors provide stronger empirical evidence to demonstrate the practical advantages of the proposed framework? We have expanded the evaluation to five datasets by adding Credit Approval, QSAR Biodegradation, WDBC, and Raisin. Across 24 comparisons at fixed $\nu$, selecting $\tau$ gives **lower mean error in 18 comparisons, higher mean AUC in 16, and higher mean F1 in 18**. Tables 1–5 in the Official Comment report the full results. For example, at $\nu=0.30$ on WDBC, error decreases from $4.92\\%$ to $3.34\\%$ and F1 increases from $92.90\\%$ to $95.32\\%$. ## Q5. > Additionally, can the authors clarify the respective contributions of Dirichlet and Gaussian weights through ablation studies? Table 6 in the Official Comment reports the weight laws selected across the five outer folds at each $\nu$. WDBC selects Dirichlet weights in all folds, whereas QSAR selects Gaussian weights in all folds at $\nu=0.50$ and $0.40$. Both cases show outperformance. The data decide via cross validation whether a more or less conservative objective is preferred. Both Dirichlet interpolation and Gaussian extrapolation contribute to this choice, with their use depending on $\nu$ and the dataset. Removing either one restricts selection to one side of the whole path and is therefore not ideal. ## References [1] Aharon Ben-Tal, Dick den Hertog, Anja De Waegenaere, Bertrand Melenberg, and Gijs Rennen. Robust Solutions of Optimization Problems Affected by Uncertain Probabilities. *Management Science*, 59(2):341–357, 2013. [2] John Duchi and Hongseok Namkoong. Variance-based Regularization with Convex Objectives. *Journal of Machine Learning Research*, 20(68):1–55, 2019. [3] R. Tyrrell Rockafellar and Stanislav Uryasev. Optimization of Conditional Value-at-Risk. *The Journal of Risk*, 2(3):21–41, 2000. [4] Hongseok Namkoong and John C. Duchi. Stochastic Gradient Methods for Distributionally Robust Optimization with $f$-divergences. In *Advances in Neural Information Processing Systems*, volume 29, pages 2208–2216, 2016.
Response to Reviewer 3
We greatly thank the reviewer for the constructive review. Below are our responses to each point.
W1.
We have simplified the notation throughout the paper and clarified the motivation. We kindly refer the reviewer to the updated paper.
Let Xθ(ξi) be the loss of model θ on observation ξi. Classic Distributionally robust optimization (DRO) accounts for uncertainty in the data distribution by minimizing the largest expected loss over an ambiguity set [1]. A candidate with nonnegative weights wi summing to one has expected loss ∑iwiXθ(ξi). With ambiguity set W, the classical DRO problem is
minθsupw∈W∑i=1nwiXθ(ξi).
A classical DRO selects the most adverse candidate in W with the inner maximization, even when we would only assign tiny probability to it.
To account for other candidates, we propose to assign a probability distribution to the candidates and aggregate their expected losses through a risk functional. This lets the learning objective account for how candidates are distributed and how strongly their adverse losses should matter.
Let W=(W1,…,Wn)∼π be random weights with ∑iWi=1. For a risk functional R, the proposed learning objective is
\qquad W\sim\pi.$$ While the math tools in Theorem 1 is standard, it leads to an important DRO interpretation of the proposed learning objective, which has not appeared in the literature. ## W2. > Although Appendix E discusses standard convergence conditions, no convergence rate or finite-iteration guarantee is provided for Algorithm 1. We have added a **finite-iteration guarantee (Theorem 17, Appendix E.6)**. Assume convex losses and regularization on a closed convex feasible set, an attained finite minimum, and bounded subgradients. Run Algorithm 1 with fresh independent Dirichlet batches, step sizes $\eta_t=(t+10)^{-3/5}$, and average the final half of the updated iterates. For any fixed batch size $M\geq1$, $$\mathbb{E}[F(\bar z_T)]-F^* \leq\frac{2(D_0+5G_F^2)}{T^{2/5}},\qquad T\geq10.$$ Here $F$ is the joint decision–threshold objective, $\bar z_T$ the returned average, $D_0$ the initial squared distance to a minimizer, and $G_F$ a subgradient bound, using the scaled norms defined in the appendix. Informally, the expected optimization error decreases at rate $O(T^{-2/5})$, even with a fixed Monte Carlo batch size. Corollary 18 gives $O(\varepsilon^{-5/2})$ iterations to reach expected error at most $\varepsilon$. The expectation is over the weight draws, with the observed data fixed. ## Q1. > What are the concrete advantages of specifying an outer risk functional and a distribution over empirical weights rather than directly defining a DRO ambiguity set? For any given risk functional $\mathcal{R}$ with envelope representation, the Dirichlet–Gaussian family provides objectives ranging from the empirical average $\frac1n\sum_{i=1}^n X_\theta(\xi_i)$, through $\mathcal{R}(X_\theta)$, to more conservative objectives. Dirichlet weights provide the interpolation and Gaussian perturbations the extrapolation. The weighted losses are ordered in convex order, giving this risk ordering under Theorem 6’s assumptions. **The classical DRO formulation does not have the interpolation-extrapolation scheme for a given risk functional.** Furthermore, we tune the parameter $\tau$ by cross-validation, so the data determine whether the objective should be more or less conservative than the original $\mathcal{R}(X_\theta)$.