- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks when KKT multipliers from a solver inside an actor-critic loop help as critic inputs. Being a deterministic function of the context , they add no Bayes information; for nested linear classes the gain, , comes from the target-aligned part of not linearly explained by the baseline features and is traded against estimation cost (Prop. 1, Thm. 1). When duals depend on the action, on-graph critic values identify the detached action gradient only up to , while total-derivative and score-function updates are invariant (Thm. 2). Experiments span synthetic checks, FQE, and full RL with Double-DQN (including ACN charging traces) and MASAC.
Critical strengths
The paper explains why giving solver duals to the critic can help even though they add no new information. It also shows that one natural way of using action-dependent duals in the policy update relies on a gradient the critic's training data cannot pin down, and identifies two update rules that avoid this. The main strengths are listed as followed:
- Insightful framing of the critic-side question. The same-information observation, , rules out the tempting "duals add information" explanation and puts the question where it belongs: learnability within a restricted class. Splitting the gain into recoverability (the residual ) and relevance () is intuitive and yields a falsifiable prediction, confirmed in Fig. 2b: no gain when either is absent.
- A practically relevant actor-side result. Theorem 2 gives the exact ambiguity set of the detached action gradient and shows that the total derivative is identified. This shows a common pattern: feeding action-dependent solver outputs to the critic and stop-gradienting through them. The -extension audit (Eqs. 34-35) is a reusable diagnostic, and the matching actor and critic hashes for Total and Score across (App. D.4.1) are a convincing check.
Critical weaknesses
My main concern is the gap between the theory and the neural or RL experiments. and the finite-sample margin are never estimated in a realistic task, so the conclusion's ``design principle'' is not operationalized, and no experiment shows a solver representation being neutral or harmful outside the Gaussian simulation (harm only at ).
Questions for authors
- Which variant were the shared RL hyperparameters developed on? Do the MASAC and queue-task conclusions hold when Context is tuned independently with the same search budget, and with substantially longer training?
- How do the FQE gains change at γ = 0.9 or 0.99?
- Is the gain at α = 0 (Fig. 7a) due to input scaling, conditioning, or optimization speed? Does it disappear with input normalization, longer training, or a wider network, and how much of the RL gains could it account for?