- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper observes that under shared parameters, the action-value baseline is not vaiance-optimal for the sum of per-decision gradient contributions, and proposes a closed-form row-wise projection correction. Experiments on three black-box tasks show variance reduction and improved learning.
Critical strengths
- The core observation is that per-decision optimality is not equal to joint optimality under shared parameters.
- Lemma 4.3 gives a clean action-wise decomposition; Theorem 4.5 reduces the correction to simple row-wise projections, avoiding an extra network or covariance solve.
- The three tasks span continuous, discrete, and combinatorial settings, and Figure 1 directly tests the decomposition.
Critical weaknesses
- The abstract promises a “variance-optimal” correction, but Theorem 4.5 only guarantees optimality within a single row’s input span. The unrestricted optimum is explicitly just a benchmark (Remark A.3). This scope should be stated in the abstract, not buried in the appendix.
- The core observation—per-decision optimality ≠ joint optimality under shared parameters—is standard. Lemma 4.3 is a telescoping identity, and Theorem 4.5 is essentially an orthogonal projection onto a one-dimensional subspace. Corollary A.4 largely recovers known results [1], [2]. The contribution is useful, but the “framework” language overstates it.
- Algorithm 1 costs model evaluations per step and grows exponentially with unless continuations are enumerated or sampled. All experiments use exact enumeration (≤625 sequences), which is not the long-horizon LLM setting the paper invokes. The sampled variant appears in only one sentence (Remark A.5), with no experiments or variance analysis.
- RELAX performs surprisingly poorly in Figure 2, with no tuning details given. It is unclear how the exact gradient is computed for MCP and JobShop; if by enumeration, it is the full expectation, not an oracle. Learning gains are also reported only qualitatively, without final performance or AUC.
- Notation is inconsistent and there are some typo errors.
[1] Action-dependent Control Variates for Policy Optimization via Stein Identity [2] Evaluating the Variance of Likelihood-Ratio Gradient Estimators
Questions for authors
See weaknesses.