- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes RL-PaO, which treats the predicted uncertain parameter as an RL action and incorporates the downstream optimizer and execution process into the environment. It is evaluated using several microgrid applications.
Critical strengths
I think the problem intuition itself is meaningful: prediction accuracy can be poorly aligned with downstream decision quality in many problems, which are the motivation for decision-focused learning. Some numerical evaluation is relevant.
Critical weaknesses
-
I am not convinced this is really an RL problem. The setup looks very close to standard decision-focused learning: predict an uncertain parameter, solve the downstream optimization problem, and evaluate the induced decision [1–3]. It is unclear what genuine sequential decision making is needed here.
-
The novelty over existing decision-focused learning is weak. The main motivationthat prediction accuracy may not align with downstream decision quality is already well established [1–3]. Prior work has also considered energy scheduling applications [1].
-
A black-box MILP does not by itself justify RL. There are already decision-focused methods that do not require differentiating through the downstream optimizer [4]. The paper does not explain why policy-gradient RL is preferable to these alternatives.
-
There are also critical issues in the MDP formulation. The paper uses continuous state and action spaces but invokes finite-state/action arguments for the existence of an optimal policy. It is also unclear why the weighted RL reward fundamentally differs from the weighted prediction/decision losses that the paper criticizes.
-
The comparison with decision-focused learning is limited. The paper includes a supervised PaO baseline based on SPO+, but the authors themselves note that SPO+ is not well suited to their constraint-side uncertainty setting. Meanwhile, other black-box or gradient-free decision-focused methods are discussed in the paper but not evaluated. Therefore, the experiments do not establish that RL provides an advantage over existing decision-focused approaches.
[1] Donti, Amos, and Kolter. Task-Based End-to-End Model Learning in Stochastic Optimization. NeurIPS, 2017. [2] Wilder, Dilkina, and Tambe. Melding the Data-Decisions Pipeline: Decision-Focused Learning for Combinatorial Optimization. AAAI, 2019. [3] Elmachtoub and Grigas. Smart “Predict, then Optimize”. Management Science, 2022. [4] Shah et al. Decision-Focused Learning without Differentiable Optimization. NeurIPS, 2022.
Questions for authors
- What is the actual sequential dependence in this problem? Does the prediction at time t affect a future physical state and later rewards? If not, why is RL needed rather than standard decision-focused learning [1–3]?
- What is the main methodological difference from existing DFL, beyond treating the prediction as an “action” and downstream cost as a “reward”?
- Could you compare against a non-RL black-box decision-focused method such as [4]? This would help show whether RL itself provides any benefit.
- Could you include more fair decision-focused learning baselines? I think there are too much relevant literature to compare.