acceptodds
Under review as a conference paper at ICLR 2027

Decision-Aware Training for Sample-Based Generative Models

Abstract

Sample-based generative models are increasingly used for probabilistic forecasting in high-stakes decision settings, yet their training objectives are blind to the decision maker’s cost structure. These models are commonly trained with strictly proper scoring rules, such as the energy score, which allocate their training signal in proportion to data density, with no awareness of where forecast errors are most costly for downstream decisions. We therefore propose decision-aware training for sample-based generative models, augmenting the energy score objective with a differentiable decision loss that directly penalises the cost incurred by acting on the model's forecast. This combined loss is theoretically grounded, as the decision loss is itself a proper scoring rule. We compute the decision loss via a differentiable optimisation layer. Its gradient concentrates in cost-sensitive regions of the output space, making the method's effects interpretable and predictable from the cost structure. We validate the method on one synthetic and two real-world tasks. In the synthetic task, the method corrects the mode weights of a learned bimodal distribution; in a wind power dispatch task, it concentrates improvements in the rare but costly tail region, and in a frost protection task, it improves how well the decision costs are anticipated. Our method yields generative models that retain full probabilistic forecasts while being better aligned with the decision maker's specific cost structure.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Probabilistic forecasting, Decision-focused learning, Scoring rules

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4tc4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a differentiable decision loss into the training process to consider the forecast errors that are most important for downstream decisions. This method can help improve the interpretability of generative models from the cost structure. The paper evaluates the proposed approach on one synthetic and two real-world tasks, showing the effectiveness of decision loss.

Critical strengths
  1. The problem is important and well motivated.
  2. The gradient analysis provide the evidence for the authors' claim.
Critical weaknesses
  1. There are already some papers introducing decision loss into generative models, such as [1-3]. However, the authors claim that "no existing decision-focused learning method operates in the sample-based setting".
  2. The paper considers a very limited baselines. Many relevant studies should be included in both discussion and experiments [1-6].
  3. In all real-world experiments, the aggregate decision costs do not have a clear improvement.
  4. The paper seems to simply introduce the decision loss in generative models.
  5. The theoretical analysis mainly focuses on the formulation rather than the meaningful guarantee.

[1] Wang, Prince, et al. "Gen-DFL: Decision-focused generative learning for robust decision making." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Zhao, Zihao, et al. "Diffusion-DFL: decision-focused diffusion models for stochastic optimization." International Conference on Learning Representations. Vol. 2026. 2026.

[3] Hemmati, Mahya, and Maryam Daryalal. "Generative Decision-Focused Learning for Multistage Stochastic Convex Programming."

[4] Lerch, Sebastian, et al. "Forecaster's dilemma: Extreme events and forecast evaluation." Statistical Science (2017): 106-127

[5] Zhao, Shengjia, et al. "Calibrating predictions to decisions: A novel approach to multi-class calibration." Advances in Neural Information Processing Systems 34 (2021): 22313-22324.

[6] Sahoo, Roshni, et al. "Reliable decisions with threshold calibration." Advances in Neural Information Processing Systems 34 (2021): 1831-1844.

Questions for authors

Please see the Weaknesses.

Reviewer_x5sKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper provides a principled approach to solving a decision task by training a sample-based probabilistic regression model for probabilistic forecasting and solving the optimization problem to obtain the optimal decision based on the sample-based forecast. To train the regression model, the authors propose using a convex-combination of an energy score and the decision loss for the task. The usual energy score component ensures the combined loss is strictly proper while the decision loss penalizes forecasting errors in a way that is consistent with the incurred cost of the subsequent selected action.

Critical strengths
  • The approach is principled and elegant.
  • The paper is clearly written and polished and a pleasure to read. There are many appendices with additional interesting details.
  • Even if the authors cover only two real-world use-cases, the detailed analysis of the results is full of interesting insights.
Critical weaknesses
  • My main concern regards the decision miscalibration metric used in this work. While the authors are inspired by [1], they evaluate this metric as the absolute difference between predicted expected cost and realized cost before averaging, rather than after, as the original work intends. As a result, the metric would penalize irreducible variability in the outcome and will not vanish, even under a perfect forecast. In my view, this undermines interpretation of this metric on which many of the conclusions rely.
  • Regardless of the point above, it seems to me that improving the decision loss would be the ultimate goal of decision-aware training. However, it is difficult to assess whether this is accomplished, and I don't find a meaningful discussion about this point in the main text (which centers mostly on the miscalibration metric):
    • For the wind power results in the main text, the losses are decomposed across three regions but the introduction of the decision loss improves the decision loss in the cut-off region at the cost of a worse score in the more frequent rated region. The aggregate outcome is only plotted in an appendix, where it looks like the decision loss is rather flat (although this is difficult to tell since 3 lambdas are plotted together making the scale of the plot too large for the variation in the curves).
    • Likewise, for the frost protection results it seems like the impact of increasing the decision loss weight seems rather small (although, again, this might be obscured by the scale of the plot).
  • While I liked the depth of the analysis on the two real-world case studies, I think having a wider range of test cases, even if not analyzed as thoroughly, would significantly improve the paper.

Note: I wanted to give this paper an accept rating but the concern expressed in the first point is preventing me from doing so.

[1] Zhao et al. Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration. NeurIPS 2021

Questions for authors

Suggestions:

  • Why not take the absolute value after averaging the cost prediction errors? Or perhaps better, use a binned version of this metric (similar to ECE)?
Reviewer_M3LyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a decision-aware training framework for sample-based generative forecasting models The method combines the energy score with a downstream decision loss defined as the realized cost of the Bayes-optimal action induced by the predicted distribution.

The framework is evaluated on implicit generative models and distributional diffusion models using a synthetic bimodal forecasting problem and two ERA5-based applications. The experiments show improved optimal-action tracking in the synthetic setting and improvements in selected cost-sensitive regions. However, aggregate decision costs remain largely unchanged in the real-world experiments, while the most consistent reported improvements concern a metric termed decision miscalibration. Moreover, the paper has two major weakness. First, its theoretical and methodological novelty is insufficiently established, with limited comparisons to closely related decision-focused learning and probabilistic forecasting methods. Second, the decision-calibration metric is inconsistent with this formal definition, and the limited improvements in aggregate decision costs raise concerns about the practical benefits and generalizability of the proposed approach.

Critical strengths

S1. Coherent framework

The paper provides a theoretically grounded approach to incorporating downstream decision costs into sample-baed generative forecasting while preserving population-level strict properness.

S2. Informative empirical analysis

The experiments and ablations provide useful insights into how decision-aware training affects predictive distributions, particularly in cost-sensitive regions, while also highlighting its limitations.

Critical weaknesses

W1. Limited theoretical and methodological novelty.

My primary concern is that the paper does not sufficiently distinguish its contributions from established and recent work in probabilistic decision-focused learning. Learning a probabilistic predictive model by differentiating cost is already established in Donti et al. Therefore, the characterization in Section 6 that existing decision-focused learning approaches target point predictions is inaccurate when applied to this work.

Moreover, decision-aware fomulation is closely related to the well-established notion of decision regret. In decision-focused learning, regret is commonly defined as the realized cost of the forecast-induced optimal action relative to an oracle action with perfect foresight. Therefore, using the realized optimal-action cost as an objective is not a fundamentally new paradigm.

In particular, Stratigakos et al. already formulate a decision-aware forecasting objective combining decision regret with CRPS. Imporantly, their decision regret is defined as the realized cost of the forecast-induced optimal action minus the perfect-foresight optimal cost. Since the latter is independent of the predictive model, minimizing this regret is equivalent to minimizing the optimal-action cost proposed in the present paper. Thus, the proposed decision loss appears to be a reformulation of an established regret-based objective.

I acknowledge that two works differ in their model parameterization. Nevertheless, since the paper positions its decision-aware training framework as a central methodological contribution, the novelty of this claim appears overstated. Extending an established regret-based objective to different generative architectures does not, by itself, constitute a fundamentally new training principle. While its implementation for sample-based generative models may offer practical value, the paper does not sufficiently distinguish such an extension from a substantive methodological advance.

The absence of a discussion of this closely related work in Section 6 further weakens the novelty justification. The authors should clarify which aspects of the proposed training methodology are genuinely novel beyond adapting an existing decision-focused objective to sample-based generative models.

W2. Insufficient comparison with closely related forecasting and decision-focused learning methods.

Although the authors include an energy-score-only baseline (wd=0w_d=0), this primarily serves as an internal ablation rather than a comparison with existing decision-aware forecasting approaches. Diffusion-DFL provides a closely related approach that trains diffusion models for downstream stochastic optimization using generated samples. Despite differences in architecture and gradient estimation, adapting this method to the same decision tasks would provide a meaningful comparison of downstream decision quality, forecasting accuracy, and computational efficiency. Without such a comparison, it is difficult to establish the practical advantages of the proposed decision-aware training framework over existing diffusion-based decision-focused learning approaches.

Furthermore, although the Introduction motivates the proposed framework through recent advances in probabilistic weather forecasting, Section 6 largely omits the corresponding forecasting literature. Existing weather forecasting approaches and their training objectives are cited in the Introduction and background but are not adequately discussed or positioned in the Related Work section. Given that the paper explicitly targets probabilistic forecasting applications, a more comprehensive discussion of this research context is necessary to clarify how the proposed decision-aware training differs from and complements established forecasting methods.

I encourage the authors to strengthen both the experimental comparisons and the related-work discussion to better establish the practical significance and positioning of the proposed framework.

W3. Insufficient evidence of practical benefits and general-purpose forecasting utility

One of the primary advantages of probabilistic forecasting is that a single predictive distribution can support multiple downstream decisions. Once a sufficiently accurate conditional distribution is learned, different decision-makers can compute their optimal actions from generated samples without retraining the forecasting model.

In contrast, the proposed approach incorporates a particular decision cost directly into the generative-model training objective. This may specialize the learned predictive distribution to a specific downstream task. Although the exact population-level objective remains strictly proper, this does not guarantee that the pratically learned distribution retains it utility across other decision problems. A direct cross-deicision evaluation would be informative. This would clarify whether method learns generally useful probabilistic forecasts or primarily task-specific decision representations.

W4. Misalignment of the decision-calibration metric with its formal definition

Section 3, Eq. (3), defines decision calibration through equality between the predicted expected decision cost and the realized decision cost in expectation.

However, the reported decision-miscalibration metric averages the absolute difference between the predicted expected cost and each individual realized cost:

\left| \mathbb{E}{Y\sim\widehat F{x_i}} [c(a_i^,Y)] -c(a_i^,y_i) \right|.$$ This metric is not a consistent measure of the calibration discrepancy defined in Eq. (3). In particular, it includes irreducible outcome variability even when the predictive distribution is perfectly calibrated. More importantly, the metric can assign a strictly better score to a systematically miscalibrated forecast than to the true distribution. For example, consider the squared-error decision cost $c(a,y)=(a-y)^2$, with the true distribution $$p=0.75\delta_0+0.25\delta_1$$ and an incorrect predictive distribution $$q=0.5\delta_0+0.5\delta_{0.5}.$$ Both distributions induce the same Bayes-optimal action, $a^*=0.25$, and therefore the same realized decision costs. Nevertheless, the expected absolute cost discrepancy used in the paper equals $0.1875$ for the true distribution and $0.125$ for the incorrect distribution. Thus, the reported metric favors the incorrect distribution despite its nonzero expected-cost calibration error. The reported improvements in decision miscalibration are central to the empirical conclusions, particulary because aggregate decision costs remain largely unchanged in the real-world experiments. In Section 5.2, the aggregate wind-power decision loss is relatively flat across decision-loss weights, while the highlighted improvements are concentrated in regions conditioned on realized wind-speed outcomes. Similarly, in Section 5.3, the aggregate frost-protection decision loss remains mostly unchanged, while the energy score can deteriorate with increasing decision-loss weight. Consequently, the current experimental results do not convincingly demonstrate substantial improvements in overall downstream decision quality. References Relevant to the Review Donti, P., Amos, B., & Kolter, J. Z. (2017). Task-based End-to-end Model Learning in Stochastic Optimization. NeurIPS. Stratigakos, A., Pineda, S., & Morales, J. M. (2025). Decision-focused linear pooling for probabilistic forecast combination. International Journal of Forecasting, 41(3), 1112–1125. Zhao, Z., et al. (2026). Diffusion-DFL: Decision-focused Diffusion Models for Stochastic Optimization. ICLR.
Questions for authors

Q1. Theoretical and methodological novelty

How does the proposed decision loss after fundamentally from the established decision-regret formulation, particularly that of Stratigakos et al.? Given that the two objectives are equivalent up to a forecast-independent term, what novel theoretical or methodological contribution does the proposed training framework offer beyond applying an existing decision-focused objective to sample-based generative models?

Q2. Comparison with existing methods

Could the authors provide a comparison with Diffusion-DFL under the same downstream decision tasks, particularly in terms of decision quality, forecasting accuracy, and computational efficiency? If a direct comparison is infeasible, could the authors clarify the key methodological differences and explain why the proposed approach is preferable? Additionally, how does the method relate to existing probabilistic weather forecasting approaches discussed in the Introduction?

Q3. Generalization across downstream decisions

How well does a decision-aware forecasting model trained for a specific cost function generalize to other downstream decision objectives without retraining? For example, could the authors evaluate models trained with different wind-power penalty parameters or frost-protection cost ratios under alternative cost functions? Such experiments would help clarify whether the proposed approach preserves general-purpose forecasting utility or primarily produces task-specific predictions.

Q4. Validity of decision-calibration evaluation

Why is the mean absolute difference between predicted expected cost and individually realized cost used to evaluate the expectation-level calibration condition defined in Eq. (3)? Given that this metric can favor a miscalibrated forecast over the true distribution, could the authors provide results using calibration metrics consistent with the formal definition? Furthermore, do the reported improvements remain significant when evaluated using aggregate downstream decision costs?

Comments on the paper

1 comment
Area_Chair_JBhYOpenReview ↗

Author-Reviewer Discussion ends in 1 day, 12 hours

Dear authors and reviewers,

The discussion period closes in about a day and a half, so this is the last chance for a direct exchange before the reviewer-AC discussion starts on Nov 19.

Authors: I have not yet seen a response to the reviews. Please post a rebuttal addressing the reviewers’ main concerns, focusing on the points most likely to affect the decision. If you update the manuscript, a short summary of the changes would help reviewers.

Reviewers: please read the authors’ response once it is posted and let us know whether your concerns have been addressed. Please update your recommendation if your assessment has changed. Even a short acknowledgment helps. If you have follow-up questions, posting them early will leave the authors time to answer.

Thank you all for keeping the discussion constructive and timely.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.