acceptodds
Withdrawn from ICLR 2027

RL-PaO: Prediction as Action in Decision Making under Uncertainty

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Decision-making under uncertainty often relies on predicted parameters, yet accurate prediction does not necessarily lead to good operational decisions. Aligning prediction with downstream optimization requires learning from the consequences of the decisions those predictions induce. We introduce RL-PaO, a reinforcement learning framework that integrates system formulation, optimization, and decision execution into a single environment. This yields a Markov decision process in which prediction is regarded as action: it shifts the environment to produce subsequent context and reward that explicitly aligns prediction error with realized cost, and learning the optimal policy does not require differentiating through the black-box solver. We evaluate RL-PaO on day-ahead energy scheduling using real historical data. On the test year, RL-PaO achieves the lowest annual cost among the non-oracle baselines, achieving on average cost reduction. Moreover, RL-PaO is capable of further analyses to provide strong interpretability both from the policy evolution perspective and the cost-accuracy trade-off.

Primary area:
optimization
Keywords:
Reinforcement Learning, Decision Making under Uncertainty, Prediction, Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_D9DCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes RL-PaO, which treats the predicted uncertain parameter as an RL action and incorporates the downstream optimizer and execution process into the environment. It is evaluated using several microgrid applications.

Critical strengths

I think the problem intuition itself is meaningful: prediction accuracy can be poorly aligned with downstream decision quality in many problems, which are the motivation for decision-focused learning. Some numerical evaluation is relevant.

Critical weaknesses
  1. I am not convinced this is really an RL problem. The setup looks very close to standard decision-focused learning: predict an uncertain parameter, solve the downstream optimization problem, and evaluate the induced decision [1–3]. It is unclear what genuine sequential decision making is needed here.

  2. The novelty over existing decision-focused learning is weak. The main motivationthat prediction accuracy may not align with downstream decision quality is already well established [1–3]. Prior work has also considered energy scheduling applications [1].

  3. A black-box MILP does not by itself justify RL. There are already decision-focused methods that do not require differentiating through the downstream optimizer [4]. The paper does not explain why policy-gradient RL is preferable to these alternatives.

  4. There are also critical issues in the MDP formulation. The paper uses continuous state and action spaces but invokes finite-state/action arguments for the existence of an optimal policy. It is also unclear why the weighted RL reward fundamentally differs from the weighted prediction/decision losses that the paper criticizes.

  5. The comparison with decision-focused learning is limited. The paper includes a supervised PaO baseline based on SPO+, but the authors themselves note that SPO+ is not well suited to their constraint-side uncertainty setting. Meanwhile, other black-box or gradient-free decision-focused methods are discussed in the paper but not evaluated. Therefore, the experiments do not establish that RL provides an advantage over existing decision-focused approaches.

[1] Donti, Amos, and Kolter. Task-Based End-to-End Model Learning in Stochastic Optimization. NeurIPS, 2017. [2] Wilder, Dilkina, and Tambe. Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization. AAAI, 2019. [3] Elmachtoub and Grigas. Smart “Predict, then Optimize”. Management Science, 2022. [4] Shah et al. Decision-Focused Learning without Differentiable Optimization. NeurIPS, 2022.

Questions for authors
  1. What is the actual sequential dependence in this problem? Does the prediction at time t affect a future physical state and later rewards? If not, why is RL needed rather than standard decision-focused learning [1–3]?
  2. What is the main methodological difference from existing DFL, beyond treating the prediction as an “action” and downstream cost as a “reward”?
  3. Could you compare against a non-RL black-box decision-focused method such as [4]? This would help show whether RL itself provides any benefit.
  4. Could you include more fair decision-focused learning baselines? I think there are too much relevant literature to compare.
Reviewer_Ga1KOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This work studies the problem of accurately estimating the parameters of a constrained optimization program when there exists uncertainty around the exact values of those parameters. The authors formulate this problem as a closed-loop reinforcement learning task where the real-world performance serves as the reward signal. The overall setup is such that: solving a constrained optimization problem gives a control strategy, the parameters of this optimization problem are outputted by an RL policy, the control strategy's cost is then used to train the RL parameter policy such that the cost is minimized by estimating the parameters more accurately. The paper shows comparison w.r.t. other non-RL approaches and establishes superior cost efficacy of their final control strategy. Their finding highlights that better cost effectiveness might occur at imperfect parameters of the optimization problem.

Critical strengths
  1. RL formulation of the control task is neat. The paper does a good job at distinguishing itself against the existing paradigms with illustrations (Fig 1).
  2. The RL training eventually shows consistently lower cost compared to the baselines.
  3. Result about imperfect parameter estimation leading to better cost effectiveness is interesting. It can have potential implications for representativeness of the constrained optimization program designed to model the task.
Critical weaknesses
  • Clarity of the writing is obscured by short names: paper defines PtO, Pb-O, PaO, DO, RO, SO at the beginning and keeps referring to them over the entire draft. As a reader, it was extremely difficult for me to keep track of what each of these mean and how they apply in general.

  • Unsupported claim in contributions: The claim in parsimonious formulation says "we prove that …". I am not sure if there is any theorem proving such a thing, or if authors want to highlight the empirical provenance, then they could rewrite this part with clear mention.

  • Lines 266-268 mentioning PPO as effective RL algorithm for offline data and convergence guarantee about global optimum: This is incorrect. The convergence guarantee is for online data setup. The statement currently sounds as if PPO is guaranteed to reach optimality in offline setup. I went through your supplementary code and you are using stable baselines3 implementation of PPO with a custom environment. This is an online data setup, online w.r.t. the policy that outputs LP variables.

  • Figure 3 on advantage of RL-PaO: The paper only measures eventual cost at convergence. What about the cost incurred during RL training? It is not accounted for when compared against other baselines. If it takes 500 plus episodes (if I understand the setup correctly, it is basically 500 days) to become competitive against other baselines, what is the utility of applying RL here? Maybe, the method should use off-policy/offline RL methods on the data collected from existing baselines and then outperform them.

  • Claim on interpretability: The paper is using single mode multi-dimensional normal policy for continuous action prediction. I am not sure how the convergence of policy to a mode eventually is more interpretable compared to other existing approaches. Why are the existing approaches not so interpretable?

  • Table 3 has no statistical error bands. The work uses only 5 random seeds.

Questions for authors
  1. Please provide more evidence for statistical significance of the results.
  2. Please experiment with off-policy/offline RL methods in order to have a clear utility.
  3. Figure 6-b text is too small to read.
  4. What is the "mean" in figure 7?

Overall, I feel that the idea is neat and potentially, has good value. I am not sure if the paper in its current form does a good job in clarity of exposition and use of RL methods for the RL formulation given in the paper. I would be happy to have a discussion with the authors about their thoughts on this.

Reviewer_W5kjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors propose to train a predictor using feedback from downstream operational outcomes. Predictions are treated as actions, while optimization, scheduling, and execution form the environment. The reward combines prediction error and realized cost, avoiding differentiation through the downstream solver. In the day-ahead energy scheduling application, the policy predicts photovoltaic generation bounds for a MILP optimizer. Table 3 reports the lowest annual cost among the non-Oracle baselines on the held-out 2019 dataset.

Critical strengths

(1). The application motivates decision-aware prediction. Prediction errors can have asymmetric operational consequences after passing through scheduling constraints. Evaluating a predictor by the costs it induces is therefore well motivated.

(2). The black-box interface has practical appeal. Training the predictor from operational feedback can retain an existing optimizer and its constraints without requiring solver derivatives. This is a useful interface for the MILP setting considered here.

(3). The experiments provide encouraging evidence in the studied setting. The evaluation uses 2016–2018 for training and a separate 2019 test year, includes several baseline families, and reports five independent random-seed runs. The policy-distribution, error–cost, and hyperparameter analyses also provide useful observations about the method's behavior.

Critical weaknesses

(1). The paper states that each RL step corresponds to one day and that training traverses historical days sequentially. However, it does not clearly explain which consequences of one day's prediction or dispatch persist into later days, or how important those dependencies are in this task. Whether the terminal storage state is carried across days is one unresolved example. Feeding the generated schedule into the next observation creates a recursive input, but its role in addressing the task's inter-day dependencies remains unclear.

I cannot identify a clear advantage of this sequential formulation over learning from downstream costs while making predictions and decisions independently for each day. The paper presents the MDP construction as a central contribution, but the reported comparisons provide no direct evidence separating the value of multistep modeling from the benefit of cost-feedback training. The results therefore support the effectiveness of the overall approach, while leaving the distinct contribution of its multistep structure unresolved.

(2). The contributions section claims to prove that the schedule contains sufficient information to infer the uncertain bounds, but I could not locate the corresponding argument in the manuscript or appendix. In particular, the sufficiency of the schedule-only observation for next-day uncertainty and the stated Markov formulation is not established. Equation (4) also remains a weighted combination of prediction error and operational cost. Maximizing this return is not generally equivalent to minimizing annual operational cost alone. The improvement of both reward and cost in Figure 2 provides empirical evidence of alignment, but does not establish the equivalence posed in RQ1 or explain why moving the composite objective into a reward resolves the trade-off criticized in supervised PaO.

(3). Figure 1 and Section 3.2 provide a high-level interaction loop, but the temporal sequence and execution rules remain incomplete. It is unclear which day's schedule forms the observation, how the first observation is generated, and what information is available when the next prediction is made. More importantly, a schedule optimized using predicted PV bounds may need adjustment when actual PV differs, yet the balancing or recourse rule used to calculate realized cost is not specified in the manuscript. These details determine what the reported cost measures and whether it represents a feasible day-ahead execution protocol.

Questions for authors

(1). What is the day-indexed observation–prediction–execution sequence, including the initial schedule and whether the terminal battery state is carried into the following day?

(2). How is the planned dispatch adjusted when predicted and realized PV differ, and how is the resulting operational cost computed?

(3). How is SPO+ adapted to uncertain constraint bounds, and what forecasting inputs and models distinguish the DO and PtO implementations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.