acceptodds
Withdrawn from ICLR 2027

Learning Advantage Function Surrogates via Convex Conjugate Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A policy trained to make per-unit decisions, such as how much of a product to order, tells us the best action but not the cost of deviating from it. That cost, the advantage function, is what downstream systems need when they have to depart from the per-unit optimum: to estimate the impact of ordering more or fewer units, to respond to a one-off vendor discount, or to satisfy a shared capacity constraint. The advantage function is what makes such trade-offs computable, yet pure policy-learning algorithms often produce the policy alone, without a critic that would yield it. We propose a method to train an advantage function surrogate through price perturbations motivated by a Fenchel-Young loss. We describe a two-stage recipe (fit the policy, then the cost model) and illustrate it on a multi-period inventory management problem.

Primary area:
reinforcement learning
Keywords:
advantage function, Fenchel-Young loss, convex conjugate, deep reinforcement learning, inventory management

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zxFvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for learning advantage-function surrogates from pretrained policies through convex conjugate matching. The method employs a Fenchel–Young-inspired objective with dual perturbations to identify the local geometry of the advantage function while preserving the original policy. The authors establish first-order optimality conditions and a policy-gradient equivalence result, and demonstrate the approach on a multi-period inventory management problem.

Critical strengths
  • The problem setting is interesting and well-motivated. Recovering decision-relevant advantage information from policy-only learning is an important problem.
  • The problem is well-formulated. The separation between policy learning and advantage surrogate learning is reasonable and sound.
Critical weaknesses
  • Theorem 3.2 establishes gradient equality only at stationary points, under a particular state-perturbation distribution and policy parameterization. This does not guarantee equivalent gradients or decisions for arbitrary downstream policies or optimization problems.
  • The current evaluation only considers one scenario, i.e., the stationary multi-period inventory management experiments; more experiments are expected.
  • Lack of comparisons with existing advantage-function approximation methods.
Questions for authors
  • Under what conditions on the perturbation distribution, surrogate family, and true action-value function can the surrogate be uniquely identified?
  • How sensitive is the learned surrogate to the quality of the pretrained policy and the choice of perturbation distribution?
  • Please add comparisons with existing advantage-function approximation methods to make the evaluation more solid.
1 comment

Thank you for the careful read and valuable feedback. As noted in the response to the first reviewer, we will update the paper based on the feedback, and then withdraw the paper in order to write a more complete version, especially in order to augment the empirical validation section.

On the Theorem (W1). We agree. Theorem (now) 4.4 holds only at stationary points and in expectation under ρθ⊗p\rho^\theta\otimes p, and does not by itself say anything about the decisions of a downstream problem; the revision now states this. To address decisions directly we added Proposition 4.1, which holds for every μ\mu and not only at stationary points: the excess loss lies between m/2m/2 and L/2L/2 times the mean squared difference between the decisions the surrogate induces and the true decisions, over the prices in the support of pp.

On experiments and comparisons (W2, W3, Q3). We acknowledge that the experiments cover a single problem and do not include comparisons with critic-based approximations of the advantage. We chose a problem whose true advantage can be derived so that the fitted surrogate can be checked against ground truth. We will update the paper with more elaborate applications in the future.

Q1: conditions for unique identification. Corollary 4.2 gives them.

Q2: sensitivity to the policy and to p(λ)p(\lambda). On the policy: the surrogate targets the advantage of the given policy, whatever its quality, and the loss does not require optimality. The anchored family does: if the policy does not minimize Qθ(s,⋅)Q^\theta(s,\cdot), the anchored condition ∇f(a∗)=0\nabla f(a^*)=0 is misspecified by exactly ∇aQθ(s,πθ(s))\nabla_a Q^\theta(s,\pi_\theta(s)), and the non-anchored family should be used (Section 5.1). On pp: Proposition 4.1 makes the dependence explicit. The guarantee covers the prices in its support, and the one-dimensional example shows that the second moment of pp sets how sharply the loss determines the curvature. We have not measured empirical sensitivity to the spread and centering of pp in this revision.

Reviewer_HNTuOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to provide a quantitatively sensitive analysis of the shadow price/hidden cost of the decisions generated by a policy. Simply learning the policy does not guarantee this level of information extraction, represented by the advantage function. The author proposed a two-stage approach by first learning the policy, then obtaining the surrogate, which is strictly convex and trained according to a chosen distribution of prices and access to the continuation value function. The idea is validated with a zero-lead-time, lost-sales inventory problem.

Critical strengths

The paper uses the quadratic example throughout the setup and experiments, which facilitates understanding.

Critical weaknesses
  1. Readability/Logic: The paper is not written in an easily understood way and requires a clearer positioning argument. From the abstract, the introduction, to the problem setup, each paragraph, though documented with subtitles/paragraph titles, does not seem connected, and the logic feels hard to follow. For example, little literature on how existing work addresses the under-determinism problem, and suddenly it jumps to "A generic consumption model", and then it jumps back to how to address the under-determinism challenge. There has definitely been a lot of related literature, but the authors did not list and compare.

  2. Applicability: Although the paper suggests "the surrogate structure can encode problem knowledge, for example, a quadratic when the consumption model solves QPs", this single setup throughout makes the reader feel limited to just such a simple, differentiable, strongly convex, quadratic structure, rather than applicable to more general function structure (Section 3.1). I personally lean towards rejection because of these major weaknesses that require a substantial rewrite of the paper draft.

Questions for authors
  1. Can you provide a more comprehensive literature review and give a better positioning of this paper? In its current form, I can't tell the novelty and unique contributions. It feels like a combination of existing works. For example, compare closely with
  • Gu et al. (2016), Continuous Deep Q-Learning with Model-based Acceleration
  • Blondel et al. (2020), Learning with Fenchel-Young Losses
  • Li, Wu, and Mao (2025), A Fenchel-Young Loss Approach to Data-Driven Inverse Optimization
  • Gupta and Zhang (2023), Data-driven decision-focused surrogate modeling
  • Berthet et al. (2020), Learning with Differentiable Perturbed Optimizers
  • Amos et al. (2017), Input Convex Neural Networks (you mentioned in the paper but no reference nor comparison)
  1. Could you list the required information ( besides the trained policy to make your framework work, e.g., counterfactual continuation costs and choice of the price distribution? Also justify if it is reasonable to obtain such information in real life.
1 comment

Thank you for the careful read and valuable feedback. As noted in the response to the first reviewer, we will thus update the paper based on the feedback, and then withdraw the paper in order to write a more complete version, especially in order to augment the empirical validation section.

On readability and logic. Point well taken. The paper has gone through a number of edits, and unfortunately, that probably shows as a disjointed exposition. We tried to address that to some extent. The introduction now runs: motivation; the consumption model, which is the object the paper is built around; how existing approaches obtain the cost of deviating from a policy and where they fall short; our approach; a list of contributions; and a roadmap.

On applicability beyond quadratics. The method requires a strongly convex surrogate family whose conjugate gradient can be evaluated and differentiated; the quadratic is the simplest member, used in the experiments because its conjugate is closed-form, but the results are not specific to it. However, our experiments do not test these shapes, and the conclusion names them as the main open question.

Q1: positioning against the listed works. Section 2 now includes a wider discussion of related work, including the suggested papers. Our contributions is a loss that needs only evaluations of QθQ^\theta at the surrogate's decisions, results tying it to decision error and identification, and a protocol that fits it to a frozen policy.

Q2: required information. Section 5.1 ("Required inputs") now lists three inputs besides the trained policy.

  • The continuation value VθV^\theta at counterfactual next states, from a critic or from rollouts under the policy. In an exo-MDP the transition and cost are known functions of the action, so the next state of a perturbed action is obtained by replaying the stored exogenous trace, with no new interaction with the system. This is the same information the upstream policy is trained with, so it is available whenever the policy is trained in this way.
  • Differentiability of the transition and cost in the action, or else a zeroth-order gradient estimator.
  • The price distribution p(λ)p(\lambda). By Corollary 4.2 it should cover the prices the consumer will impose: historical shadow prices of the coupling constraints, the range of vendor discounts, or, as in our experiments, a range set by the cost parameters. The guarantee of Proposition 4.1 holds on its support and nowhere else, so this is the main modeling decision.
Reviewer_Q1YbOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method for training an advantage function surrogate with the Fenchel-Young loss. This method enables learning not only the optimal action but also the curvature of the advantage function, which is useful in settings such as constrained inventory management. For practical training, the authors propose a two-stage formulation that first trains the policy and then trains the surrogate. Empirical results on a simple multi-period inventory management problem are provided to validate the method.

Critical strengths
  1. The methodology of using the Fenchel-Young loss for advantage surrogate training is novel to the best of my knowledge.
  2. The overall structure of the paper is clear.
Critical weaknesses
  1. The empirical validation is incomplete. Only experiments on a single simple multi-period inventory management problem are included and the experiments lack sufficient baselines. The shape function is parametrized by only two simple predefined forms.
  2. The paper lacks discussion of the relationship between the proposed method and alternative approaches to the same downstream decision problems. For example, to account for different storage-capacity limits, one can train the policy with vanilla actor-critic method under different storage-capacity parameters (with the capacity parameter as an input to the critic and actor).
Questions for authors
  1. What are the benefits of the proposed method over the actor-critic method mentioned in Weakness 2?
  2. Does this method apply in stochastic policy case?
  3. Minor issues. The citation format should be corrected (should be \citep in many cases). In the "anchored parametrization" paragraph of Section 3.1, the symbol uu should be explicitly defined.
1 comment

Thank you for the thoughtful feedback. We will try to address the comments, acknowledging that to fully answer the valid criticism about the limited empirical validation would require more work than time permits. We will thus update the paper based on the feedback, and then withdraw the paper in order to write a more complete version.

On the empirical validation (W1). The contribution of the paper is the loss and what it guarantees, and the inventory problem serves as an illustration. We chose it because the true advantage of the trained policy can be derived in closed form, allowing us to us check the fitted surrogates against the true object, which is not possible on benchmarks where QθQ^\theta is unknown. We agree that one problem and two quadratic shapes are a narrow test, and will update the paper in the future with more substantial examples. Note however that the theory does not depend on the quadratic: Proposition 4.1 holds for any strongly convex family whose conjugate gradient can be evaluated and differentiated.

On conditioning an actor-critic on the capacity parameter (W2, Q1). This is a valid alternative, and Section 2 ("Conditioning the policy on the constraint") now discusses it. An earlier version of the paper actually included a short discussion, and we now reintroduced it. When the constraint is shared across many products, the input relevant to each product after dualization is the shared shadow price λ\lambda, and buying policies that take this price as an input have been trained and coordinated in this way (Eisenach et al., 2024; Wang et al., 2026). The surrogate also induces a price-indexed policy, a(λ,s;μ)=∇fμ∗(λ;s)a(\lambda,s;\mu)=\nabla f_\mu^*(\lambda;s), with three differences.

  • The price must be part of the training of a price-indexed policy, so the approach does not apply to a policy already trained without it. Our surrogate is fit to a frozen, possibly deployed, policy, which is left unchanged.
  • A price-indexed policy is consistent with a convex cost only if it is the gradient of a convex function of λ\lambda. A generic network does not have this property, and imposing it requires input-convex or gradient-of-convex architectures; the surrogate has it by construction.
  • Consumers with nonlinear objectives gg need a cost, not a price-to-action map. The surrogate provides the cost fμf_\mu, and its dual function is differentiable, so a shared price can be set by dual gradient ascent.

On stochastic policies (Q2). Section 5.1 now addresses this. The answer is mixed. The loss requires only evaluations of QθQ^\theta, which are defined for stochastic policies, and the surrogate's perturbed action remains deterministic. What does not carry over is the anchored family: since Ea∼πAθ(s,a)=0\mathbb E_{a\sim\pi}A^\theta(s,a)=0, the advantage takes negative values that an anchored surrogate fμ≥0f_\mu\ge0 cannot represent, so the general (non-anchored) family should then be used.

Minor issues. Citations now use \citep/\citet as appropriate. In the anchored parameterization, u=a−a∗(s;θ)\mathbf u = \mathbf a - \mathbf a^*(\mathbf s;\theta) is now defined as the deviation from the anchor.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.