- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Following previous work on stochastic dominance guarantees for imitation learning, this work focuses on guaranteeing multivariate second order stochastic dominance. SSD is a property of learned policies over the expert demonstrator. This is defined for a given/assumed reward class and using the corresponding feature distributions of the learned and expert policies.
This paper develops an imitation learning algorithm that constructs an imitation policy whose feature distribution satisfies (up to some slackness) SSD over the reference feature distribution. For both left-sided and right-sided SSD, the algorithm is empirically validated over a diverse set of environments, comprising discrete and continuous state/action sets.
Critical strengths
- this paper is well motivated, clear and well written
- the proposed method is very simple
- the algorithm is evaluated on a diverse set of environments
Critical weaknesses
- The definition of the target distribution is unclear to me. Specifically, the right-hand side of equation 14 has free indices in i and j in the definition of P_target.
- Proposition 3 and Corollary 4 are given without proof. Recommendation for acceptance can only be given if the results are verifiable.
- There seems to be a typo in the VaR definition on page 2, where the stated CVaR definition is integrating the right tail. Did you mean to define \operatorname{VaR}_{1-\alpha}(R) = \inf_{t \in \mathbb{R}}\left\{t : \mathbb{P}(R \leq t) \geq \alpha\right\\}?
Minor:
- the paper is inconsistent on the notation of the target distribution, it uses both P_target and P_trg.
Questions for authors
- Table 1: it seems that R-SSD does better than L-SSD even for LCVaR objectives. How would you explain this behavior?
- Table 2: L-SSD is not doing better than FSD for LCVaR metrics. In fact, FSD beats L-SSD on every LCVaR columns. How do you explain this?
- Are the metrics computed on shared rollouts? In that case, why is it possible to have LCVaR_001 > LCVaR_005 in the first and fourth row of Table 1? Please clarify how the metrics are computed in your work.
The LP is used to obtain a target probability distribution. Then that target probability distribution is used as a training signal to push the imitator's distribution towards the target distribution. For example for left-sided SSD, we use (13) to obtain γ\* and ϵ\*, then construct target probability using (14), and then (15) for optimizing the policy. The role of the LP is to tell us how to construct the target distribution.
The LP in (13) starts with a uniform probability distribution over demonstrations. It transfers probability mass from mixtures of demonstrations to each imitator subject to that imitator being better (component-wise) than the mixture of demos (in other words, dominating that mixture of demos, such that the feature vector is component-wise greater than weighted average of demonstrator feature vectors). If some mass cannot be pushed to imitators, the demonstrator samples keep that mass and that is represented by slack variable ϵj. The last constraint ensures that sum of mass on each demonstrator, and the mass `transferred' to imitator samples from each particular demonstrator remains 1/M. Minimizing the slack variable ϵj achieves the goal of pushing as much probability mass as possible to imitators.
The LP gives us the optimal ϵ\* and γ\*. Now Equation (14) comes in. To construct a target distribution, we consider the union of both demonstrator samples and imitator samples. Each of the imitator samples ξi get mass ∑jγi,j∗, while the remaining mass that remains on demonstrators ξ~j is ϵj. This constructs the target distribution on union of demonstrator samples and imitator samples.
Finally, this target probability is used as a surrogate reward to update policy using Policy Gradient / REINFORCE style update.
We will update Equation (14) in the final version to explicitly clarify the target probability distribution being assigned to both demonstrator and imitator samples.
Yes the policy improvement is definitely related to REINFORCE but instead of using the true returns of trajectories or rewards of actions; the target probability distribution is used as a policy improvement signal. We note this in the paper on page 5 under equation (15).
Another difference from how REINFORCE is typically used in practice is that we need an entire set of trajectories to compute target probabilities. Hence this `surrogate' reward of target probability mass is assigned at the trajectory-level not at the state-action level.
It is not an advantage function because we do not use a baseline to compute each trajectory's advantage relative to that baseline. Instead, we use the absolute (not relative) target probability of each trajectory.
We are working under the premise that demonstrations are not always optimal, and are noisy. So our goal is not to exactly copy the demonstrator, but to outperform them. There are many existing works that aim to do this, under different assumptions of unknown reward functions, or by obtaining ranked preferences of trajectories from the user.
We aim to build an imitator that outperforms the demonstrator for any unknown reward function as long as that reward function is a) monotonically non-decreasing and b) concave with respect to reward features.
In Tables 1 and 2, the column heading names the metric being measured, and the row heading names the algorithm being used. So that particular cell, "MaxEnt-IRL with LCVAR$_{0.01}$" tells us when using MaxEnt IRL what is the value of Left-sided Conditional Value at Risk at α=0.01 of rollouts from a policy trained using MaxEnt IRL.