- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
This paper is a theoretical analysis on offline imitation learning algorithms where one learn from state-action pairs from both a preferred policy and a non-preferred policy, but no succeeding states are observed; in this case, one can learn preference scores for state-action pairs, but under a KL-regularized objective, the retrieved policy occupancy is not necessarily a valid one that fits Bellman flow constraint, since such constraint involves transition kernel information that we have no access to. The paper conducts theoretical analysis on this problem, and gives sufficient and necessary condition on the identifiability of occupancy. The paper further proposes SCOR, a way to evaluate occupancy certainty with finite samples. The paper conduct experiments to verify the conclusions and effectiveness of SCOR.
Critical strengths
-
This paper shed light on a novel issue: the difference between score identification and target occupancy identification, which is a valid conceptual contribution.
-
The conversion of the optimization problem under unknown dynamics to a geometric problem is interesting, and the theoretical analysis into the particular problem is thorough (with a sufficient and necessary condition and a way to determine uncertainty under finite examples).
Critical weaknesses
While the reviewer instruction asks me to identify one or two critical weaknesses, I feel this paper has more than two critical weaknesses that I need to list here.
- The paper is hard to follow. Take the first section for example:
a) Why does offline imitation learning often use demonstrations from contrasting behavior sources (line 34)? While this is often the case (e.g. expert and task-agnostic data in many DIstribution Correction Estimation (DICE) methods [1, 2, 3]), there is no intuition provided behind this practice and can make readers confusing - one can also simply use only preferred policy for behavior cloning.
b) What is "conditioning" in line 38, "conditioning removes state-visitation differences"? If it is "conditioned score on the current state" (line 37), then why does it "removes state information" (line 39), and how does it "isolate action choice" (line 38)? While I can somewhat infer that it is a decomposition from state-action joint distribution to state and state-conditioned action distribution, detailed discussion on these terms without explanation makes the paragraph difficult to read.
c) What is "joint contrast" in line 42? I think "state-action score / density ratio" is a term easier to understand.
d) "This occurs because a score and its target are different objects. a score ranks state-action pairs, whereas an occupancy describes how often a policy visits them along a discounted trajectory." First, I would suggest adding an intuitive example here for better understanding; second, the word "ranking" is not accurate as it is not about determining the order of state-action pair, but about ratio of preference.
e) What does "reweightings" in line 50 mean, and what does "unknown flow constraints" in line 51 mean? What does "optimizer" mean in line 68? These are discussed before proper introduction, and the term "unknown flow constraints" is not explained throughout the paper (Bellman flow constraint is explained later but "unknown" is not explained). The writing of paragraph 3 assumes the reader clearly know that "even one can determine the ratio between optimal occupancy and that of non-preferred policy's, the policy could be non-reachable due to unknown dynamics; we need to optimize KL-based objective which needs dynamics", which is not explained in this section.
- The motivation of this paper on studying source-only interface is unclear.
a) One very important use of non-preferred dataset is to induce the underlying dynamics of the environment. It is the very basis of all model-based offline IL methods and also implicitly applied in model-free offline IL methods to calculate value function [2]. The paper does not explain why the source-only interface is important.
b) In Sec. 3, both datasets are defined as "trajectory datasets", which should contain transition information. What is the motivation behind intentionally discarding transition information if it is already available? Could the author provide an example?
c) Practically, when I get the ratio between non-preferred policy and optimal policy, I can simply conduct weighted behavior cloning to retrieve the preferred policy [1, 2]. In this case, the ambiguity of the policy's state(-action) occupancy does not matter, since it is impractical to get an 100% accurate function estimator for every state-action pair in high-dimensional MDPs such as MuJoCo (which is even not so high-dimensional in today's RL). Although this policy does not necessarily optimize the paper's occupancy-KL objective, it is unclear why identifying the optimum of this particular objective is more necessary or beneficial for practical imitation learning than recovering the preferred policy.
-
The paper does not contain any experiment, either tabular or continuous MDP, on the actual performance of imitation learning. Thus, we do not know the actual effect of transition kernel uncertainty on the imitation learning performance of the resulting policies. The practical implication of the proposed identifiability analysis remain unclear. One experiment that the authors can do is to fix the policy ratio and evaluate policy of the performance under compatible transition kernels.
-
I think this paper does not conduct sufficient discussion on the literature. This paper only cites 17 papers in total with a short paragraph of related work, and does not conduct discussion on the offline DICE line of work (though safeDICE is cited) [1-4] which uses both preferred and non-preferred data. For example, DemoDICE [4] has a similar objective of an expert occupancy matching objective with an additional KL regularizer.
References
[1] https://arxiv.org/abs/2202.13536
[2] https://arxiv.org/abs/2202.02433
Questions for authors
- The theoretical assumption that both source occupancies are strictly positive is a rather strong one - especially for expert policy which often seldom visits most suboptimal states. Is there a way to somewhat loose this assumption?
2 comments
Thank you for your detailed response. I increased my score from 1 to 2. I am still leaning on rejection as the insights/empirical evidences on actual agent performance and on real-life applications are relatively limited. This is also pointed out by the other two reviewers (ncuP: "I am a bit confused about why the paper just throws out the next state information"; "unclear how the algorithm truly scales to setups of practical importance"; TEKK: "it is unclear why this restriction is necessary in practical settings"; "without validation on realistic imitation learning benchmarks").
Thanks for your detailed response. The introduction section is now much easier to read (though it is still hard to read for someone less familiar with the literature - I would recommend using a less compact teaser figure), and I am willing to increase my "presentation score" from 1 to 2. I also appreciate the authors' effort to add performance-related experiments. Before I reconsider my rating, I have a follow-up question: Can the author provide examples of real-life scenario where people only have access to "tables of discounted state-action counts" instead of either trajectories or state-action-next state transition counts, and where identifying the transition core is more important than optimizing the agent's performance on a MDP? It seems that to accurately log state-action counts, the MDP has to be a tabular one (if it is continuous, especially with noise, all state-action pairs will be unique, and the space required to store unique states and actions will be identical to that storing the trajectories).
Minor Issues:
Line 90, "contracts at rate n−1/2" -> O(n−1/2) or Ω(n−1/2) depending on the actual results.
The connection of the objective in Sec. 4 to the DICE methods should also appear in Sec. 4.
I would still encourage the authors to more broadly examine and discuss related works (possibly with the help of AI agents).