- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for learning advantage-function surrogates from pretrained policies through convex conjugate matching. The method employs a Fenchel–Young-inspired objective with dual perturbations to identify the local geometry of the advantage function while preserving the original policy. The authors establish first-order optimality conditions and a policy-gradient equivalence result, and demonstrate the approach on a multi-period inventory management problem.
Critical strengths
- The problem setting is interesting and well-motivated. Recovering decision-relevant advantage information from policy-only learning is an important problem.
- The problem is well-formulated. The separation between policy learning and advantage surrogate learning is reasonable and sound.
Critical weaknesses
- Theorem 3.2 establishes gradient equality only at stationary points, under a particular state-perturbation distribution and policy parameterization. This does not guarantee equivalent gradients or decisions for arbitrary downstream policies or optimization problems.
- The current evaluation only considers one scenario, i.e., the stationary multi-period inventory management experiments; more experiments are expected.
- Lack of comparisons with existing advantage-function approximation methods.
Questions for authors
- Under what conditions on the perturbation distribution, surrogate family, and true action-value function can the surrogate be uniquely identified?
- How sensitive is the learned surrogate to the quality of the pretrained policy and the choice of perturbation distribution?
- Please add comparisons with existing advantage-function approximation methods to make the evaluation more solid.
Thank you for the careful read and valuable feedback. As noted in the response to the first reviewer, we will update the paper based on the feedback, and then withdraw the paper in order to write a more complete version, especially in order to augment the empirical validation section.
On the Theorem (W1). We agree. Theorem (now) 4.4 holds only at stationary points and in expectation under ρθ⊗p, and does not by itself say anything about the decisions of a downstream problem; the revision now states this. To address decisions directly we added Proposition 4.1, which holds for every μ and not only at stationary points: the excess loss lies between m/2 and L/2 times the mean squared difference between the decisions the surrogate induces and the true decisions, over the prices in the support of p.
On experiments and comparisons (W2, W3, Q3). We acknowledge that the experiments cover a single problem and do not include comparisons with critic-based approximations of the advantage. We chose a problem whose true advantage can be derived so that the fitted surrogate can be checked against ground truth. We will update the paper with more elaborate applications in the future.
Q1: conditions for unique identification. Corollary 4.2 gives them.
Q2: sensitivity to the policy and to p(λ). On the policy: the surrogate targets the advantage of the given policy, whatever its quality, and the loss does not require optimality. The anchored family does: if the policy does not minimize Qθ(s,⋅), the anchored condition ∇f(a∗)=0 is misspecified by exactly ∇aQθ(s,πθ(s)), and the non-anchored family should be used (Section 5.1). On p: Proposition 4.1 makes the dependence explicit. The guarantee covers the prices in its support, and the one-dimensional example shows that the second moment of p sets how sharply the loss determines the curvature. We have not measured empirical sensitivity to the spread and centering of p in this revision.