acceptodds
Under review as a conference paper at ICLR 2027

Target Identification in Offline Imitation Learning: The Effect of State Conditioning

Abstract

We study reward-free offline imitation learning with logs from a preferred and a comparison behavior source. Their differences can be used to score candidate behavior. The resulting target is the state-action visitation distribution that maximizes this score while remaining feasible under the environment's dynamics and close to the logged behavior. We ask whether source visitation frequencies alone determine that target when the audit does not use linked transitions. A joint score uses both where the sources go and what actions they choose there; conditioning on state removes the first kind of information. We show that this removal can make the target ambiguous: the same two source occupancies uniquely determine every joint target in a two-state example, yet permit different conditional targets under different compatible dynamics. Under positivity and when the sources do not span all feasible flow directions, we prove that a Kullback-Leibler (KL) regularized target is uniquely determined exactly when its unconstrained reweighting is a nonnegative occupancy in the affine span of the sources. For finite samples, State-Conditional Occupancy Reweighting (SCOR) returns a target estimate and a confidence ball covering every target compatible with the true source occupancies, without estimating transitions. Sampling uncertainty shrinks with more data for identified targets with bounded affine coefficients, while uncertainty caused by indistinguishable dynamics can persist. Experiments across 30 source geometries examine these claims and the value of signed affine combinations. The guarantees concern the target specified by the learning objective, not task return.

Primary area:
reinforcement learning
Keywords:
offline imitation learning, imitation learning, identifiability, occupancy measures, state conditioning, KL regularization, partial identification, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_L54GOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper is a theoretical analysis on offline imitation learning algorithms where one learn from state-action pairs from both a preferred policy and a non-preferred policy, but no succeeding states are observed; in this case, one can learn preference scores for state-action pairs, but under a KL-regularized objective, the retrieved policy occupancy is not necessarily a valid one that fits Bellman flow constraint, since such constraint involves transition kernel information that we have no access to. The paper conducts theoretical analysis on this problem, and gives sufficient and necessary condition on the identifiability of occupancy. The paper further proposes SCOR, a way to evaluate occupancy certainty with finite samples. The paper conduct experiments to verify the conclusions and effectiveness of SCOR.

Critical strengths
  1. This paper shed light on a novel issue: the difference between score identification and target occupancy identification, which is a valid conceptual contribution.

  2. The conversion of the optimization problem under unknown dynamics to a geometric problem is interesting, and the theoretical analysis into the particular problem is thorough (with a sufficient and necessary condition and a way to determine uncertainty under finite examples).

Critical weaknesses

While the reviewer instruction asks me to identify one or two critical weaknesses, I feel this paper has more than two critical weaknesses that I need to list here.

  1. The paper is hard to follow. Take the first section for example:

a) Why does offline imitation learning often use demonstrations from contrasting behavior sources (line 34)? While this is often the case (e.g. expert and task-agnostic data in many DIstribution Correction Estimation (DICE) methods [1, 2, 3]), there is no intuition provided behind this practice and can make readers confusing - one can also simply use only preferred policy for behavior cloning.

b) What is "conditioning" in line 38, "conditioning removes state-visitation differences"? If it is "conditioned score on the current state" (line 37), then why does it "removes state information" (line 39), and how does it "isolate action choice" (line 38)? While I can somewhat infer that it is a decomposition from state-action joint distribution to state and state-conditioned action distribution, detailed discussion on these terms without explanation makes the paragraph difficult to read.

c) What is "joint contrast" in line 42? I think "state-action score / density ratio" is a term easier to understand.

d) "This occurs because a score and its target are different objects. a score ranks state-action pairs, whereas an occupancy describes how often a policy visits them along a discounted trajectory." First, I would suggest adding an intuitive example here for better understanding; second, the word "ranking" is not accurate as it is not about determining the order of state-action pair, but about ratio of preference.

e) What does "reweightings" in line 50 mean, and what does "unknown flow constraints" in line 51 mean? What does "optimizer" mean in line 68? These are discussed before proper introduction, and the term "unknown flow constraints" is not explained throughout the paper (Bellman flow constraint is explained later but "unknown" is not explained). The writing of paragraph 3 assumes the reader clearly know that "even one can determine the ratio between optimal occupancy and that of non-preferred policy's, the policy could be non-reachable due to unknown dynamics; we need to optimize KL-based objective which needs dynamics", which is not explained in this section.

  1. The motivation of this paper on studying source-only interface is unclear.

a) One very important use of non-preferred dataset is to induce the underlying dynamics of the environment. It is the very basis of all model-based offline IL methods and also implicitly applied in model-free offline IL methods to calculate value function [2]. The paper does not explain why the source-only interface is important.

b) In Sec. 3, both datasets are defined as "trajectory datasets", which should contain transition information. What is the motivation behind intentionally discarding transition information if it is already available? Could the author provide an example?

c) Practically, when I get the ratio between non-preferred policy and optimal policy, I can simply conduct weighted behavior cloning to retrieve the preferred policy [1, 2]. In this case, the ambiguity of the policy's state(-action) occupancy does not matter, since it is impractical to get an 100% accurate function estimator for every state-action pair in high-dimensional MDPs such as MuJoCo (which is even not so high-dimensional in today's RL). Although this policy does not necessarily optimize the paper's occupancy-KL objective, it is unclear why identifying the optimum of this particular objective is more necessary or beneficial for practical imitation learning than recovering the preferred policy.

  1. The paper does not contain any experiment, either tabular or continuous MDP, on the actual performance of imitation learning. Thus, we do not know the actual effect of transition kernel uncertainty on the imitation learning performance of the resulting policies. The practical implication of the proposed identifiability analysis remain unclear. One experiment that the authors can do is to fix the policy ratio and evaluate policy of the performance under compatible transition kernels.

  2. I think this paper does not conduct sufficient discussion on the literature. This paper only cites 17 papers in total with a short paragraph of related work, and does not conduct discussion on the offline DICE line of work (though safeDICE is cited) [1-4] which uses both preferred and non-preferred data. For example, DemoDICE [4] has a similar objective of an expert occupancy matching objective with an additional KL regularizer.

References

[1] https://arxiv.org/abs/2202.13536

[2] https://arxiv.org/abs/2202.02433

[3] https://arxiv.org/abs/2311.01331

[4] https://iclr.cc/virtual/2022/poster/6025

Questions for authors
  1. The theoretical assumption that both source occupancies are strictly positive is a rather strong one - especially for expert policy which often seldom visits most suboptimal states. Is there a way to somewhat loose this assumption?
2 comments

Thanks for your detailed response. The introduction section is now much easier to read (though it is still hard to read for someone less familiar with the literature - I would recommend using a less compact teaser figure), and I am willing to increase my "presentation score" from 1 to 2. I also appreciate the authors' effort to add performance-related experiments. Before I reconsider my rating, I have a follow-up question: Can the author provide examples of real-life scenario where people only have access to "tables of discounted state-action counts" instead of either trajectories or state-action-next state transition counts, and where identifying the transition core is more important than optimizing the agent's performance on a MDP? It seems that to accurately log state-action counts, the MDP has to be a tabular one (if it is continuous, especially with noise, all state-action pairs will be unique, and the space required to store unique states and actions will be identical to that storing the trajectories).

Minor Issues:

Line 90, "contracts at rate n−1/2n^{-1/2}" -> O(n−1/2)O(n^{-1/2}) or Ω(n−1/2)\Omega(n^{-1/2}) depending on the actual results.

The connection of the objective in Sec. 4 to the DICE methods should also appear in Sec. 4.

I would still encourage the authors to more broadly examine and discuss related works (possibly with the help of AI agents).

Thank you for your detailed response. I increased my score from 1 to 2. I am still leaning on rejection as the insights/empirical evidences on actual agent performance and on real-life applications are relatively limited. This is also pointed out by the other two reviewers (ncuP: "I am a bit confused about why the paper just throws out the next state information"; "unclear how the algorithm truly scales to setups of practical importance"; TEKK: "it is unclear why this restriction is necessary in practical settings"; "without validation on realistic imitation learning benchmarks").

Reviewer_TEKKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates target identifiability in offline imitation learning, distinguishing between learning preference scores and identifying the corresponding target behavior. The authors provide theoretical conditions for target identification and introduce a method for estimating target occupancies with finite-sample confidence guarantees.

Critical strengths

1.The paper investigates a fundamental problem in offline imitation learning by distinguishing preference estimation from target behavior identification. 2.The theoretical analysis is relatively rigorous, with clear mathematical formulations and finite-sample guarantees.

Critical weaknesses

1.The authors intentionally exclude state transition information, although offline imitation learning datasets typically contain complete trajectories. Since transition information can help resolve uncertainty about environment dynamics, it is unclear why this restriction is necessary in practical settings. The authors should better justify the motivation and applicability of this assumption. 2. The proposed framework assumes finite state-action spaces and strictly positive source occupancies. These assumptions are difficult to satisfy in high-dimensional continuous control environments, where data coverage is often limited. Further analysis is needed to establish whether the theoretical results can extend to more realistic settings. 3.Mathematically identifying a unique target occupancy does not necessarily imply better imitation policies or higher task returns. The paper lacks sufficient evidence connecting its theoretical findings to improvements in practical policy learning. 4. The experiments are limited to synthetic environments, without validation on realistic imitation learning benchmarks. Moreover, the proposed confidence bounds may become uninformative in larger state-action spaces or under insufficient data coverage. More challenging experiments are needed to demonstrate the practical usefulness and scalability of the approach.

Questions for authors

Please refer to the Weaknesses section for my questions and concerns.

Reviewer_ncuPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents an offline imitation learning approach with access to samples (state,action) pairs from preferred and comparison policy (loosely interpreted as good and bad policy). Crucially, we are not provided access to next state (implying the dynamics arent well specified). The paper elaborates on conditions that can help us achieve a near optimal policy and presents an algorithm "SCOR" towards recovering such a solution, and presents associated empirical results on some discrete mdp setups.

Critical strengths

The paper is mathematically pretty precise about their claims, also presenting cases where extrapolating beyond prescriptions from the good policy still continues to yield valid results that outperform the convex combinations. It also presents clear hyper-parameter free guarantees.

Critical weaknesses

-- The paper is written in a highly technical and nontransparent manner, preventing a clear understanding of their claims and contributions. The exposition needs to be made more accessible for broader understanding of their contributions.

-- i am a bit confused about why the paper just throws out the next state information, this seems like a very common piece of information available particularly in offline setups where one might want to leverage model based approaches even more than in other contexts.

-- Assumptions of full coverage (theorem 5.1) is highly restrictive.

-- Similarly, the necessary part of theorem 5.1 requires one action to transition to every state which breaks under common mdp setups e.g. gridworld / other locally connected mdp setups etc.

-- one step per episode sampling (section 5.3) introduces horizon (1/ (1-\gamma)) type blowups in sample efficiency.

-- experiments are fairly limited in scope, unclear how the algorithm truly scales to setups of practical importance.

Questions for authors

none, most of the issues are highlighted in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.