- Rating
- 4(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies an important and practical problem in reinforcement learning policy evaluation: how to collect online data so that an off-policy estimator has low variance even when the true environment dynamics differ from the simulator used for training. Existing behavior policy search methods optimize the data-collection policy for a fixed transition model, so the variance reduction can degrade under sim-to-real transition mismatch. The paper deals with this as a minimax problem: an adversarial transition model chooses dynamics that maximize evaluation variance, and the behavior policy chooses actions to minimize that worst-case variance. It proposes DRVG, a double-loop gradient algorithm. The inner loop does gradient ascent on the transition parameters; the outer loop does projected gradient descent on the behavior-policy parameters. The paper also adopts an optional KL regularization term to keep the adversarial transition close to the original simulator. The main theoretical contributions are the analytical transition-gradient formulas and convergence result for the inner adversarial optimization, followed by a global convergence guarantee for the outer DRVG algorithm. Importantly, the global result is proved under a linear-softmax behavior-policy parameterization, bounded features and importance ratios, and compact convex parameter sets; under these assumptions the worst-case variance objective becomes convex in the behavior-policy parameters and DRVG achieves an expected convergence rate with some additional terms. The empirical validation shows that DRVG has the smallest increase in variance under its own adversarial transition and also achieves the lowest variance among several methods under a shared perturbed transition.
Critical strengths
1. The paper addressed an important problem in behavior policy search. Existing BPS methods optimize the data-collection policy for a fixed transition model, but the paper recognizes that the quantity being optimized is ultimately evaluated in an environment with shifted dynamics. Formulating the problem as a minimax problem is neat: the adversary represents the transition uncertainty, while the behavior policy is explicitly optimized for worst-case evaluation variance.
2. The transition-gradient derivation is a good methodological contribution. A key technical step is showing that the gradient of the variance with respect to the transition parameters can be written in terms of the transition score and moments of the OPE estimator. This makes adversarial transition optimization possible directly from sampled trajectories and the paper derives the expression for a general OPE estimator rather than being tied intrinsically to one particular estimator. The resulting Algorithm 1 also uses sample splitting to construct an unbiased estimate of the product-of-expectations term.
Critical weaknesses
The global convergence guarantee depends on an inner-loop accuracy condition that the proposed algorithm does not establish. Specifically, Theorem 5.4 assumes that, at each outer iteration, the adversarial transition is an -optimal solution to the inner maximization problem. It is unclear whether the practical algorithm can achieve this inner-loop guarantee. Consequently, the paper's main theoretical guarantee does not fully guarantee the behavior of its proposed algorithm.
Questions for authors
Q1. How does the practical inner-loop algorithm satisfy the accuracy assumption required by the global convergence theorem? Theorem 5.4 assumes that the inner maximization returns a -optimal adversarial transition at every outer iteration. However, Algorithm 1 and Lemma 4.2 establish convergence to a stationary point, which seem not to guarantee global optimality.
Q2. How are the compactness and feasibility assumptions on the transition parameters enforced in Algorithm 1? The convergence analysis assumes a compact transition parameter space , along with smoothness and bounded-derivative conditions for the transition model. However, Algorithm 1 seems to update the parameter using an unconstrained gradient-ascent step, without projection onto the space . Could the authors clarify how the transition parameters are kept within the theoretically assumed feasible set during optimization? Specifically, does the implementation impose parameter bounds, apply projection or clipping?
Q3. How realistic is the convergence rate when its constants depend strongly on the horizon ? Theorem 5.4 establishes a theoretical guarantee, but the constants depend on the horizon . In particular, scales as . This seems huge. Could the authors clarify the regimes where this guarantee provides a realistic sample-complexity bound, rather than only an asymptotic convergence rate? For example, can the dependence on the horizon be improved under certain additional conditions?
2 comments
Many thanks for taking the time to review our responses and revised manuscript! We are glad that our clarifications and the improved horizon-dependence analysis addressed your concerns. We sincerely appreciate your thoughtful feedback and your reconsideration of our paper!
I would like to thank the authors for addressing my questions. The authors gave detailed reply to all three questions of mine, together with modification to the paper draft. I find the authors' reply directly answered all my question. Especially for Q3 where I have concerns about the strong dependence on T will make the bound poor in long horizon environments, the authors showed that the dependence can be improved without extra mathematical assumption. Overall, I appreciate the rebuttal and thus I raise my score.