acceptodds
Withdrawn from ICLR 2027

Should You Train for Decisions? A Decision-Adequacy Test and Stein Shrinkage for Predict-then-Optimize

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A cost model for an optimization problem can be fitted by least squares and then optimized (two-stage) or trained on the decisions it induces (decision-focused). Which is better depends on an unknown misspecification, so the two are mixed by cross-validation or by hand, with no guarantee. We ask instead whether the model is decision-adequate: would decision-focused training change its decisions? We test this with a Hausman-type statistic, the gap between the two fits in the geometry of the decision loss rather than the Wald geometry, which overweights directions that change no decision. Smoothing the SPO+ loss gives both fits a closed-form covariance, and the same gap is the unknown coefficient of the risk along the path between them, so one statistic says whether to move and how far: a closed-form James-Stein weight that, asymptotically in the risk of that geometry, is never much worse than the better fit and, under a regime condition, beats the decision-focused fit. In preregistered experiments the test has size 0.051 under correct specification and rejects in most splits on two electricity markets where decision-focused training cuts regret by up to a factor of six. In practice this gives a p-value for decision adequacy and, for methods that interpolate between the fits, a closed-form step that requires no refits and no prior knowledge of which fit is better. Every failed criterion is reported: the shrinkage gain is small, the rule trails cross-validation where the smoothing scale is coarse, and the test over-rejects under heavy tails at small samples and reads skewed noise as misspecification.

Primary area:
optimization
Keywords:
decision-focused learning, SPO+, James-Stein shrinkage, predict-then-optimize, model misspecification, hypothesis testing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_FLGQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study investigates the differences in agreement between the two methodologies of Least-Squares Estimation and Optimization (ETO) and Decision-Focused Estimation; and presents an approach to combine the two methods using a principle-based interpolation method. An Entropically Smoothed SPO+ loss is presented which maintains Fisher Consistency under symmetric noise, and can be differentiated with respect to the estimated parameters. This allows for both Sandwich Covariance Estimation and a Quadratic Test Statistic to detect when there is disagreement between the estimators. The same statistic is utilized to select a Closed Form Interpolation Weight. Asymptotic Calibration, Oracle Results, Dominance Results, and SURE Safe Last Layer Variant Variants (JS-W and Cross-Fitted) are all provided in the paper. Experimental evaluations have been conducted on a variety of data types such as Synthetic Data, Combinatorial Optimization Data, Electricity Market Data, and Neural Prediction Data. These evaluations demonstrate that tests are generally well-calibrated and interpolation is beneficial.

Critical strengths

The single most significant methodological advance in this study was using the identical quadratic measure to quantify the difference in estimator performance and to estimate the unknown variance (or discrepancy) in the local path risk. By smoothing both components of SPO+ it is possible to maintain the same symmetric consistency properties associated with SPO+, yet enable estimation of the joint covariance of these components. The analysis of the data-dependent nature of the shrinkage weight using a Stein correction, as opposed to simply replacing the data dependent weight with a random weight within a fixed-weight risk formula, is another important feature.

Furthermore, the differences in the SURE dominance criterion and the wider range of conditions under which the safer constants apply, adds to the informational content of the statistical model, as opposed to a purely ad-hoc interpolation heuristic. Once the two fits and the covariance terms have been determined, evaluating the resulting weight is computationally inexpensive.

Critical weaknesses

The question in the paper is whether decision-focused training will alter the decisions made by the trained model. The paper repeatedly refers to the rejection of the hypothesis (that decision-focused training will not alter the decisions) as an indication that the model or last layer should be moved.

However, the formal null hypothesis for this study is that there are equal population parameters. The authors acknowledge, as stated in Section 2 and Remark S8-H, that equal population parameters do not necessarily mean that the decisions made by the two models will be the same; positive scaling may result in different parameter values but no change in decision, and asymmetrical error distributions may lead to the rejection of the null hypothesis while ETO is already optimal for making decisions.

Moreover, Proposition S7-d supports calibration with a correctly specified conditional mean and symmetrically distributed conditional errors. Calibration does not support the null hypothesis of equal population parameters.

Additionally, the authors conclude their discussion by stating that they have no level guarantees for adequate models with asymmetrical errors or an inaccurately specified mean. As such, the reported p-value has limited applicability in the larger context that motivated the paper.

Questions for authors

What is the exact conclusion that practitioners will be able to make based upon rejection (of their model) in addition to disagreements regarding the ETO/Smoothed-SPO+ population target?

Specifically, what additional assumptions are required to allow one to conclude that the decision-making process needs to be altered or that decision-based training may help improve it? What is the distinction between a formal parameter-equality null hypothesis, a formal equality of hard decisions null hypothesis, and the fact there is no way to achieve a reduction in regret?

In addition, Proposition S7-d states that there is no general level guarantee for certain distributions which satisfy parameter equality. Does this mean that the authors believe they have developed a calibrated test within the correctly specified symmetric model class and not a calibrated test across the entire decision-adequacy null?

Reviewer_7uMDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper considers the problem of deciding whether a given task decision task over estimated parameters (i.e. routing over distances estimated based on contextual information) should be tackled via Prediction Focused Learning (PFL) or Decision Focused Learning (DFL). The former trains the parameter estimator for maximum likelihood, the latter for minimal decision regret (by actively calling a solver for the decision problem at training time).

The authors focus on Mean Squared Error for the PFL and the SPO+ loss for DFL, and restrict their attention to a "classical" DFL setting, where both the estimator and the problem are linear, and the estimated parameters appears as coefficient of the cost function in the decision model.

The paper proposed to choose the most reliable method via a rigorous statistical test. High reliability is achieved by fitting both approaches multiple times, and by relying on an additional smoothing technique for SPO+ (to ensure a meaningful Hessian). The authors also provide a criterion (based on variance minimization) to choose a convex combination of the parameter vectors obtained via the two fits.

Critical strengths

The authors correctly build on the observation that DFL methods, even in conditions where they should have superior performance, they may fail to do so (for example due to sample scarcity, drift, or distortion of the actual regret landscape induced by the used of surrogate losses such as SPO+.

The proposed test is very rigorous, to the best of my ability to check.

The proposed criterion for "hybridizing" PFL and DFL via a convex combination is also sound.

The discussion on treating the last layer of Neural Network as a linear predictor is appreciated, though it does not represent a core contribution.

Critical weaknesses

My main concern about this work is that, for the sake of increasing rigor, it seems to sacrifice computational scalability and -- more in general -- practicality. This is apparent in a few key design choices, which I'll comment in the next paragraphs.

Training a predictive model for minimal decision cost is a costly process by itself, since it requires to repeatedly solve constrained optimization problems that are frequently NP-hard. PFL is comparatively much cheaper; therefore, a method to choose between the two would in principle be very useful.

The proposed approach, however, requires fitting both a PFL and DFL predictor (specifically, the corresponding weights), meaning that it can be used to decide between the two only once substantial computational costs have already been incurred.

In addition to that, the chosen statistical test requires a meaningful Hessian, which the authors obtain for the SPO+ loss by smoothing, in particular via Gibbs weights. The exact process requires enumerating all vertices of the feasible polytope, which are exponential in the number of decision variables in the general case. Switching to incomplete enumeration would accelerate the process, but also introduce noise. As a side-note, Gibbs smoothing would also further distort the regret landscape, skewing the position of DFL local optima w.r.t. the true regret landscape.

Other steps of the process introduce further computational costs, though those appear to be optional in some case (like the 199 used by the asymptotically exact calibration).

I can see how the approach could still be useful to decide which settings (pairs of distributions and decision problems) lend themselves to DFL or PFL, though even such an approach would still have issues in case of distribution drift.

Overall, it is my opinion that -- unfortunately -- the drawbacks of the method greatly outweigh it potential benefit, in the vast majority of practical settings.

In addition to this limitation, the paper has additional weaknesses. For example, no baseline except for simple cross-validation is considered; even cross-validation receives little attention and comments in the text seems to suggest a performance comparable to that of the proposed test. It would seem like another non-parametric statistical test on the expected regret for the two approaches would be a much simpler (if perhaps less reliable) alternative -- and benefit from the fact that regret is a scalar, rather then a vector.

Finally, the text has been likely heavy edited through AI tools. This is of course not a problem per se, but the approach resulted in this case is a presentation heavy with jargon, excessively compact sentences, overly technical, unexplained or even ambiguous terminology.

Questions for authors

Can you clarify how the two predictors should be refitted in the proposed method?

In which practical settings would you expect the method to provide a definite benefit?

Reviewer_AAdTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper fails to properly define what problem it addresses. It has no conclusions. Let's look at the beginning of the abstract: "A cost model for an optimization problem can be fitted by least squares and then optimized (two-stage) or trained on the decisions it induces (decision-focused). Which is better depends on an unknown misspecification, so the two are mixed by cross-validation or by hand, with no guarantee. We ask instead whether the model is decision-adequate: would decision-focused training change its decisions?" Probably written by AI, these sentences are unclear in setting up the objectives of the paper. On the other hand, they make many claims, unsubstantiated. The abstract follows with "We test this .....", but what is "this"?

Critical strengths

None.

Critical weaknesses

The paper is not providing the reader with any insight.

It is not at all clear what the Experiments in Table 1. and 2. and Figure 2. are supposed to demonstrate, what conclusion they are supporting.

Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.