acceptodds
Under review as a conference paper at ICLR 2027

When Do Optimization Duals Help Reinforcement Learning?

Abstract

In reinforcement learning with embedded optimization, inner solvers return Karush–Kuhn–Tucker (KKT) dual variables that provide structured abstractions of constraint and resource pressure induced by the current decision context, yet how such representations affect actor–critic learning remains unclear. We characterize when such representations improve critic learning and, when action-dependent, how they affect actor-gradient identifiability. Since the solver representation is a deterministic function of the complete optimization context , explicitly providing it to the critic adds no Bayes information. Nevertheless, the added representation can expose target-relevant solver structure that is difficult for the baseline class to recover. For nested linear or frozen-feature classes, we derive an exact fixed-design finite-sample risk decomposition balancing captured structural signal against added estimation cost, with a separate Gaussian reference crossover and a bridge to fixed-policy value learning. On the actor side, we show that the detached action partial derivative is generally not identified by on-graph critic values, with its ambiguity confined to . Matched total-derivative and score-function interfaces are instead extension-invariant. Experiments spanning controlled diagnostics, fixed policy value learning, and full actor–critic control support the predicted critic-side learnability trade-offs and actor-side gradient-identifiability effects.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Actor-Critic Methods, KKT Dual Variables, Embedded Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mQ7MOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when KKT multipliers from a solver inside an actor-critic loop help as critic inputs. Being a deterministic function of the context X=(s,a)X = (s, a), they add no Bayes information; for nested linear classes the gain, γ⊤ΣRγ\gamma^\top \Sigma_R \gamma, comes from the target-aligned part of ZZ not linearly explained by the baseline features and is traded against estimation cost (Prop. 1, Thm. 1). When duals depend on the action, on-graph critic values identify the detached action gradient only up to Range(J⊤)\text{Range}(J^\top), while total-derivative and score-function updates are invariant (Thm. 2). Experiments span synthetic checks, FQE, and full RL with Double-DQN (including ACN charging traces) and MASAC.

Critical strengths

The paper explains why giving solver duals to the critic can help even though they add no new information. It also shows that one natural way of using action-dependent duals in the policy update relies on a gradient the critic's training data cannot pin down, and identifies two update rules that avoid this. The main strengths are listed as followed:

  • Insightful framing of the critic-side question. The same-information observation, σ(X,Z)=σ(X)\sigma(X, Z) = \sigma(X), rules out the tempting "duals add information" explanation and puts the question where it belongs: learnability within a restricted class. Splitting the gain into recoverability (the residual RR) and relevance (γ\gamma) is intuitive and yields a falsifiable prediction, confirmed in Fig. 2b: no gain when either is absent.
  • A practically relevant actor-side result. Theorem 2 gives the exact ambiguity set of the detached action gradient and shows that the total derivative is identified. This shows a common pattern: feeding action-dependent solver outputs to the critic and stop-gradienting through them. The κ\kappa-extension audit (Eqs. 34-35) is a reusable diagnostic, and the matching actor and critic hashes for Total and Score across κ\kappa (App. D.4.1) are a convincing check.
Critical weaknesses

My main concern is the gap between the theory and the neural or RL experiments. Δstruct\Delta_{\text{struct}} and the finite-sample margin are never estimated in a realistic task, so the conclusion's ``design principle'' is not operationalized, and no experiment shows a solver representation being neutral or harmful outside the Gaussian simulation (harm only at n⩽16n \leqslant 16).

Questions for authors
  • Which variant were the shared RL hyperparameters developed on? Do the MASAC and queue-task conclusions hold when Context is tuned independently with the same search budget, and with substantially longer training?
  • How do the FQE gains change at γ = 0.9 or 0.99?
  • Is the gain at α = 0 (Fig. 7a) due to input scaling, conditioning, or optimization speed? Does it disappear with input normalization, longer training, or a wider network, and how much of the RL gains could it account for?
Reviewer_By5AOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the use of dual features from embedded optimization solvers as critic inputs. It analyzes approximation gains and estimation costs for fixed feature classes, with a connection to fixed-policy value estimation. It also characterizes detached action-gradient ambiguity for action-dependent features and identifies extension-invariant actor updates. Experiments cover prediction, FQE, gradient diagnostics, and online control.

Critical strengths
  1. The actor analysis provides a concrete diagnostic. The authors characterize how critics can agree at all solver-consistent inputs while producing different detached action gradients, and test update interfaces through interventions that preserve these values.

  2. The assumptions and experimental controls are carefully specified. The paper distinguishes the scope of its theoretical settings, includes random/shuffled features, primal solutions, and active-set controls, and reports cases where dual features are not the best representation.

Critical weaknesses

The core critic theory does not sufficiently explain when dual variables help. The risk comparisons in Sections 3.2.1–3.2.3 do not use properties of dual variables or RL. For example, replacing the dual features with any deterministic mapping Z=g(X)Z=g(X) satisfying the same assumptions leaves the main analysis intact: additional features reduce approximation error when they capture target components outside the original linear space, with this gain weighed against additional estimation cost. The benefit conditions thus depend on the relationship between the added representation and the unknown target, without further deriving when optimization and reward structures make these conditions hold. The sensitivity interpretation in Appendix A and the Bellman/FQE error bounds in Section 3.2.4 do not establish this connection either. Although the paper quantifies the benefits of general feature augmentation, I find its answer to the central question of when optimization duals help RL insufficient. This is the main reason for my weak-rejection recommendation.

Questions for authors
  1. Core theoretical contribution: Which results connect the sensitivity or constraint-pressure properties of dual variables to the target alignment needed for long-term value prediction? What guidance do these results provide on when to use dual features, beyond general feature-augmentation analysis?

  2. Neural critic mechanisms: For end-to-end neural critics, which experiments support the analyzed mechanism, and which gains might instead arise mainly from the optimization or conditioning effects discussed in Appendix D.1?

Reviewer_BxH2OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper considers a reinforcement learning setting where the policy action determines the parameters of an optimization problem, whose solution is then used in the environment. The authors ask whether supplying the resulting dual variables to a value function, alongside the state and action, improves prediction even when the duals provide no additional information. For linear value functions, the authors formally show that the benefit of additional features must outweigh the extra estimation error stemming from learning additional weights. They also compare the resulting policy updates (and gradients) using stop-gradient and total-derivative approaches (through the deterministic policy gradient method and reparameterization), and score-function estimation with a baseline (through the policy gradient method).

Critical strengths

(S1) The authors examine the question at hand both theoretically and empirically. I particularly appreciate the first result (Theorem 1, Corollary 1, and Figure 2c), which identifies when additional features can be useful by connecting the number of original features, the number of new linearly independent features, the target noise variance, the sample size, and the improvement in the best achievable prediction.

Critical weaknesses

(W1) I do not find that Theorem 2 (and the following corollary) provides new insight beyond applying known results (Anders et al., 2020) to value functions with auxiliary inputs. Moreover, although the first result establishes when additional features may help and the subsequent analysis connects this to iterative value function learning, these results are generic and not specific to optimizer duals. Hence, I feel like the theory addresses the question "when additional features help RL," rather than optimizer duals specifically.

(W2) (Minor) I had difficulty drawing a clear practical conclusion from the results. In particular, it remains unclear how the results should guide the decision to include dual features or the choice of a policy-gradient approach in a given RL problem with optimizer dynamics.

Questions for authors

(Q1) Have you experimented with different objective functions that yield the same primal solution but different dual variables, while keeping the RL rewards unchanged? How sensitive is value function to these changes?

(Q2) What practical conclusion should we draw from the results about when to include dual features if we have them in a value function?

(Q3) To what extent does the first theoretical result explain the gains observed with nonlinear value functions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.