acceptodds
Under review as a conference paper at ICLR 2027

Offline Quality-Diversity

Abstract

Offline reinforcement learning methods learn a single policy from a static dataset, but in doing so fail to leverage the behavioral diversity the dataset may contain. Meanwhile, Quality-Diversity (QD) methods produce rich archives of diverse high-performing behaviors, but require online environment interactions, which poses a fundamental barrier when trial-and-error is costly or unsafe. In this paper, we try to reach the best of both worlds: the same dataset that yields a single offline RL policy could, in principle, yield an entire repertoire that QD methods could find. We introduce Offline Quality-Diversity (Offline QD), a new problem setting in which the goal is to recover a full behavioral repertoire over a given descriptor space from a static dataset, without any environment interaction during training. We propose three independent methods that each solve this differently: successor features that act as a proxy for a policy's achieved descriptor; a learned world model that lets a standard QD algorithm search entirely in imagined rollouts; and a behavior-conditioned decision transformer that recovers an archive through sequence modeling alone. Experiments show that Offline QD recovers a substantial fraction of online QD performance on tasks where exploration is easy, and substantially surpasses online QD on tasks where random exploration is hard but the offline dataset provides more diversity. We further show that offline archives serve as a strong initialization for online search, turning environments where online QD struggles from scratch into tractable ones. Our results suggest that offline datasets already contain far more behavioral diversity than standard offline RL methods extract, opening a new direction for QD research beyond online exploration.

Primary area:
reinforcement learning
Keywords:
Offline Reinforcement Learning, Quality Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_R5DpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Offline Quality‑Diversity, a new reinforcement learning problem setting. Unlike traditional online Quality Diversity, which needs real‑environment interaction to evaluate behaviors, Offline QD aims to build a full QD policy archive only from static offline datasets, with no environment access during training. Given predefined behavioral descriptors, it seeks high performance policies that match each target behavior. Since existing offline methods require extra online rollouts to map onto target descriptors, they cannot solve Offline QD. To preliminarily verify the tractability of the problem, the authors adapt three established offline RL paradigms to tackle it. Experiments show these methods work across locomotion, navigation and manipulation tasks. All three methods have clear inherent limitations from out‑of‑distribution risks, model drift or dataset coverage. However, the analysis of the challenges unique to this setting remains somewhat limited.

Critical strengths
  1. Offline-QDAC leverages successor features (SF) as a proxy for behavioral compliance and is grounded in theoretical analysis. Despite a gap between theory and practical implementation arising from replacing the policy’s state distribution with the dataset distribution, the theoretical framework still helps us understand the method’s regime of validity.

  2. To address the issue that prior offline QD methods cannot be deployed in a truly fully offline fashion, the paper defines a more stringent offline QD problem formulation. This represents the paper’s primary contribution, and the work presents several basic solutions tailored to this new problem setting.

  3. The experimental validation is relatively comprehensive. It verifies that the proposed problem formulation is indeed tractable under several common experimental setups, and outperforms online algorithms in certain cases.

Critical weaknesses
  1. If the SF proxy compliance signal steps into out-of-distribution (OOD) regions, it becomes unreliable. The authors did not try to offer any effective safeguards.

  2. The three proposed solutions mainly adapt three existing paradigms to the new problem. The paper lacks sufficient analysis of the novel challenges introduced by this setting and does not provide targeted innovative ideas to address them.

Questions for authors

In the standard offline RL setting, OOD extrapolation error primarily concerns scalar rewards. In the offline QD setting, however, the BD becomes a vector, and the corresponding SF function outputs a vector as well. This raises the question: is controlling OOD extrapolation error in this scenario more challenging than controlling extrapolation error for the Q-function in conventional offline RL?

4 comments
AuthorsOpenReview ↗

Response to Reviewer R5Dp (1/2): OOD safeguards

We thank the reviewer for the constructive review, and for recognizing the Offline QD problem as our primary contribution, the theoretical grounding of the SF proxy, and the comprehensive experimental validation. In response, we (i) detail the safeguards against OOD actions and evaluate two stronger ones, AWR and in-sample SF learning (W1, Tables R1-R2), (ii) gather the challenges specific to Offline QD in Section 6 (W2), and (iii) discuss what makes extrapolation error harder to control for successor features (Q1). All numbers refer to the revised version; changes are shown in blue.


W1 (OOD safeguards). We agree. IQL's in-sample training protects the critic targets, but the actor and the SF targets query actions generated by the policy. Four mechanisms limit extrapolation in practice:

  • Critic: QQ and VV are trained in-sample with IQL [5], so their targets never use out-of-distribution actions.
  • Gate: the reward gradient is only active where the policy is already compliant, which restricts Q-maximization to those states.
  • Successor features: they are bounded by ∥ϕ∥/(1−γ)\Vert \phi\Vert /(1-\gamma) and learned by policy evaluation, without a max operator, so their errors are not amplified the way Q-learning's are.
  • Behavioral constraint: on Antmaze and Cube, the actor is regularized toward dataset actions with a BC term, following standard OGBench practice [1, 2], which limits how far the SF is queried outside the data.

Effect of the BC term. The constraint is essential on the OGBench tasks, but it reduces performance on locomotion (Table R1). We therefore apply it only to Antmaze and Cube.

Table R1: Effect of the BC term on Offline QDAC, averaged over datasets per environment (mean ± std, 5 seeds). Table 16 in the revised paper.

VariantHalfcheetahWalkerAntmazeCube
Without BC1701±4886±21287±27155±21
With BC (α\alpha = 0.3)1243±2652±9622±58261±36

Alternatives to our design. We also compare three variants, each replacing one component of Offline QDAC (App. E.6, Table 15 of the revised version):

  • TD3 critic: removes the conservative in-sample critic.
  • AWR actor [4]: restricts policy extraction to dataset actions, weighted by advantage.
  • ψ\psi-IQL: learns the successor features in-sample, mirroring IQL. We learn an action-independent function E(s,z)E(s,z) by expectile regression on dataset actions, weighting more heavily those whose successor features are closer to the target:

LE=E(s,a)∼D,z[∣τ−1[u<0]∣∥ψ(s,a,z)−E(s,z)∥22],u=∥(1−γ)E(s,z)−z∥−∥(1−γ)ψ(s,a,z)−z∥,\mathcal{L}_E = \mathbb{E}_{(s,a)\sim\mathcal{D}, z}\Big[ \big|\tau - \mathbb{1}[u < 0]\big| \big\Vert \psi(s,a,z) - E(s,z)\big\Vert _2^2\Big], \qquad u = \big\Vert (1-\gamma)E(s,z) - z\big\Vert - \big\Vert (1-\gamma)\psi(s,a,z) - z\big\Vert ,

and bootstrap the successor features through it, ψ(s,a,z)←ϕ(s)+γE(s′,z)\psi(s,a,z) \leftarrow \phi(s)+\gamma E(s',z), so that no policy action is queried. As τ→1\tau\to1, E(s,z)E(s,z) approaches the successor features of the most compliant dataset action, ψ(s,a⋆,z)\psi(s,a^\star,z) with $a^\star = \arg\min_{a \in \mathrm{supp} \mathcal{D}(\cdot|s)} \Vert (1-\gamma)\psi(s,a,z) - z\Vert $.

Table R2: Offline QDAC with alternative policy extraction and SF learning, averaged over datasets per environment (mean ± std, 5 seeds). Table 15 in the revised paper.

CriticActorSF learningHalfcheetahWalkerAntmazeCube
TD3 (no conservatism)Q-maximizationTD with policy actions1252±5484±24523±65226±31
IQLAWRTD with policy actions1422±3473±16564±61209±22
IQLQ-maximizationψ\psi-IQL (in-sample expectile)1197±4271±18514±35192±32
IQLQ-maximizationTD with policy actions (ours)1701±4886±21622±58261±36

Our design performs best in all four environments. Removing the conservative critic (TD3) lowers performance, except on Walker, where all variants overlap within one standard deviation. AWR is consistently weaker, in line with prior findings that weighted regression extracts less from a learned value than gradient-based extraction [3]. ψ\psi-IQL performs worst in all environments; we explain why in our answer to Q1.

Finally, the disagreement between the two heads of our SF network could provide an uncertainty estimate to penalize the actor. We mention this direction in Section 6 of the revised version.

AuthorsOpenReview ↗

Response to Reviewer R5Dp (2/2): Offline QD challenges, SF extrapolation

W2 (novelty and Offline QD challenges). We agree that our three methods adapt existing paradigms. However, we believe the paper and this rebuttal identify several challenges that do not arise in standard offline RL, together with how we address each. In the revised version, we gather them in Section 6 ("Challenges and Limitations"):

  • Behavior cannot be verified offline. Checking whether a policy achieves its target descriptor requires a rollout for every target in the descriptor space. Our successor-feature surrogate provides an offline signal, and Corollary 1 makes its dependence on dataset coverage explicit.
  • Data must cover the descriptor space. In offline RL we need data near high-return behavior, but in Offline QD we also need data across all target behaviors. Dataset quality thus has two competing axes: App. E.4 shows that QD score peaks at an intermediate mix of fitness-oriented and diversity-oriented data.
  • Descriptors are trajectory-level. Offline goal-conditioned RL reuses data by relabeling any future state as a goal. A QD descriptor characterizes a whole episode, so this relabeling does not directly apply. Successor features address this for additive descriptors by predicting the future discounted average of features.
  • Fitness and compliance conflict. A weighted sum lets fitness override compliance; our gate enforces compliance first. A weighted goal-plus-fitness baseline (GCIQL + fitness) reaches a lower QD score than Offline QDAC (527 vs. 622 on Antmaze, 211 vs. 261 on Cube).
  • Imagined search must cover full episodes. Without replanning from real states, world-model errors compound over the horizon, a structural limitation rather than a capacity issue (App. E.9).

We see this characterization as a starting point for methods designed specifically for Offline QD.


Q1 (extrapolation error for successor features). In our view, the difficulty does not come from the SF being a vector, but from what the SF must represent. The same difficulty would arise with a one-dimensional feature.

Why in-sample learning works for QQ but not for the SF.

  • QQ can be learned from dataset actions only. IQL estimates the value of the best behavior in the data. This only needs actions present in the data, so no unseen action is queried.
  • The SF cannot. The SF must describe the current policy πθ\pi_\theta, because it measures whether this policy reaches its target zz. Learned from dataset actions only, it describes the behaviors in the data instead of πθ\pi_\theta. To evaluate πθ\pi_\theta, the SF targets must use actions of πθ\pi_\theta, and these actions can be out of distribution.
  • Evidence. ψ\psi-IQL (W1) learns the SF in-sample, as IQL does for QQ. It avoids unseen actions entirely, yet performs worst in Table R2 (514 vs. 622 on Antmaze), since its SF do not describe the current policy (App. E.6).

Why SF errors are easier to control in one respect. The SF update has no max over actions, and the true SF is bounded by ∥ϕ∥/(1−γ)\Vert \phi\Vert /(1-\gamma). SF errors are therefore not amplified by a max operator, as Q-learning errors are. The remaining errors are limited by the safeguards of W1, in particular the BC term on Antmaze and Cube.

[1] S. Park, K. Frans, B. Eysenbach, S. Levine. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.

[2] S. Fujimoto, S. S. Gu. A Minimalist Approach to Offline Reinforcement Learning. NeurIPS 2021.

[3] S. Park, K. Frans, S. Levine, A. Kumar. Is Value Learning Really the Main Bottleneck in Offline RL? NeurIPS 2024.

[4] X. B. Peng, A. Kumar, G. Zhang, S. Levine. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv:1910.00177, 2019.

[5] I. Kostrikov, A. Nair, S. Levine. Offline Reinforcement Learning with Implicit Q-Learning. ICLR 2022.

AuthorsOpenReview ↗

Follow-up on W2: a mechanism specific to Offline QD

Regarding W2 (lack of targeted ideas for the challenges of the setting), our new global comment describes a mechanism designed for one such challenge: estimating, from offline data only, whether a policy will reach a target descriptor that is not additive, such as a final position or a range of motion. The new critic fixes the failure of Offline QDAC on the range descriptor (487 → 1896), and clearly improves it on Antmaze Explore (+60%) and Cube Play (+36%). We would be grateful to know whether this addresses the reviewer's concern.

Reviewer_R5DpOpenReview ↗

Official Review of Submission11423 by Reviewer R5Dp

Thank you to the authors for the considerable effort they have devoted to responding to the review questions. I also partly agree with several other reviewers regarding the limited methodological novelty. If the formulation of a new problem such as offline QD is to be treated as an important contribution, then the unique new challenges that such a new problem faces need to be articulated much more clearly. Although the authors do enumerate some relevant challenges in their response, challenges such as "Behavior cannot be verified offline", or "Data must cover the descriptor space", are not problems that any algorithm can solve — and the offline data available in practice is often difficult to control deliberately. I therefore believe that the new problems and new challenges in offline QD that are genuinely solvable by algorithms are indeed still not as clearly delineated as the core challenge identified for offline RL at the time, namely OOD extrapolation error. Overall, most reviewers agree that this problem is worth exploring, and the authors have indeed been the first to do a substantial amount of foundational work in this direction. I maintain my rating unchanged.

Reviewer_8vfyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Offline Quality-Diversity (Offline QD), a setting in which the objective is to recover a repertoire of diverse, high-performing policies from a fixed offline dataset without additional environment interactions during training. The authors investigate three approaches: Offline QDAC based on successor features and offline RL, JEPA-ME based on learned world models, and BD-DT based on behavior-conditioned sequence modeling. Experiments across locomotion, navigation, and manipulation tasks demonstrate that offline datasets can support the construction of QD archives and that the resulting archives can provide useful initializations for subsequent online QD optimization.

While the problem is interesting and practically motivated, I have several significant concerns regarding the technical novelty, experimental comparisons, and generality of the proposed approaches.

Critical strengths

The paper studies a meaningful problem at the intersection of offline RL and quality-diversity optimization. Recovering diverse behaviors without online interactions is potentially valuable.

The authors explore three different methodological approaches and provide extensive empirical analyses, including investigations of dataset quality, behavioral compliance, and offline-to-online transfer.

Critical weaknesses
  1. Limited methodological novelty.

The three proposed approaches largely build on existing algorithms without introducing fundamentally new learning mechanisms.

Offline QDAC adapts the existing QDAC framework by combining successor-feature-based behavioral compliance with an offline RL backbone.

JEPA-ME combines a learned latent dynamics model with standard MAP-Elites search.

BD-DT combines return conditioning from Decision Transformer with behavior conditioning already explored in QD-Transformer.

While these adaptations are reasonable, the individual technical contributions appear relatively incremental. The main novelty therefore lies in the problem formulation rather than in the proposed algorithms. I am not fully convinced that the paper establishes sufficient methodological advances beyond these combinations.

  1. The offline-versus-online comparisons do not adequately isolate the benefits of the proposed algorithms.

One of the central empirical claims is that Offline QD substantially outperforms online QD in hard-exploration environments, with improvements approaching 30×.

However, this comparison involves an important asymmetry: the offline approaches have access to pre-collected datasets containing potentially informative and diverse trajectories, whereas the online methods must discover promising behaviors from scratch.

In particular, some datasets are explicitly constructed from trajectories generated by MAP-Elites, including expert datasets collected from converged QD archives. Consequently, the offline methods benefit from behavioral information that may have required substantial environment interactions to obtain.

Although the authors include stronger online QD baselines in Appendix E.1, these comparisons do not resolve the fundamental asymmetry in prior information.

The results demonstrate that leveraging informative offline data can be advantageous, but they do not establish that the proposed algorithms are intrinsically more effective than alternative approaches under comparable data-access conditions. A clearer accounting of dataset collection costs and comparisons with data-informed baselines would substantially strengthen the claims.

  1. Insufficient comparison with alternative offline learning approaches.

Given the proposed problem formulation, offline goal-conditioned RL, reward-conditioned imitation learning, and related multi-behavior offline RL approaches are particularly relevant.

The paper includes a comparison against GCIQL in Appendix E.2, but evaluates only behavioral compliance rather than the joint quality-diversity objective. Notably, GCIQL achieves better compliance at strict thresholds on the evaluated tasks.

Although the authors explain the difference between goal-conditioned RL and Offline QD, this does not fully address whether stronger reward-aware, behavior-conditioned offline learning baselines could achieve comparable QD performance.

The lack of such comparisons makes it difficult to assess how much of the reported performance comes from the proposed Offline QD mechanisms rather than from established conditional offline policy learning techniques.

  1. Limited generality and significant failure cases.

The three approaches exhibit substantial limitations:

JEPA-ME largely fails on complex, long-horizon environments, achieving zero or near-zero QD scores in several settings.

Offline QDAC relies on a successor-feature-based behavioral compliance surrogate that is most naturally suited to additive behavioral descriptors.

BD-DT is strongly dependent on dataset quality and struggles to generalize beyond demonstrated behaviors.

Moreover, all experiments are conducted in simulation with low-dimensional behavioral descriptor spaces. The results therefore provide preliminary evidence of feasibility rather than convincing support for a broadly applicable Offline QD framework.

  1. Gap between the theoretical analysis and practical implementation.

The theoretical compliance bound relies on expectations under the stationary state distribution induced by the learned policy. However, the practical algorithm replaces this distribution with the offline dataset distribution, requiring additional coverage assumptions.

Furthermore, the theoretical setting involves long-run average descriptors, while several experimental tasks employ finite-horizon or final-state descriptors that are not directly covered by the main guarantee.

The empirical validation in Appendix E.9 reports only moderate correlation between the surrogate compliance error and the actual error (Pearson correlation of 0.69), with some cases of underestimation.

Thus, the theoretical analysis provides limited assurance about behavioral compliance in the practical offline setting.

Questions for authors

How would the conclusions change if the environment interactions used to generate the offline datasets were accounted for in comparisons with online QD methods?

Can the authors compare against stronger offline goal-conditioned or behavior-conditioned RL baselines that optimize both behavioral compliance and fitness?

What is the main algorithmic contribution beyond adapting existing successor-feature, world-model, and Decision Transformer methods to the Offline QD setting?

How robust is Offline QDAC when the offline dataset does not sufficiently cover the state distribution induced by the learned policies?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8vfy (1/3): novelty, online QD with the same data

We thank the reviewer for the detailed feedback, and for recognizing Offline QD as a meaningful problem and our empirical analyses as extensive. In response, we (i) add data-informed online QD baselines (W2, Table R1), (ii) evaluate GCIQL on the full QD objective and add two reward-aware, behavior-conditioned baselines (W3, Table R2), and (iii) clarify the scope of our theory (W5). All numbers refer to the revised version; changes are shown in blue.


W1 (novelty). We agree that our main contribution is the problem formulation, together with evidence that the problem is tractable, and we now state this more clearly in the contributions of the revised version (Section 1). Building on established paradigms was a deliberate choice: as the first study on Offline QD, our aim is to identify in which setup each proposed solution performs well and fails. We believe this gives a basis for more specialized methods in the future. We also gather the challenges specific to Offline QD, and how our methods address them, in Section 6 of the revised version.

However, combining existing methods into a working Offline QD method is not straightforward. The most direct ways of combining existing techniques fall short:

  • Combining an offline RL backbone with a compliance reward (IQL + combined reward) reaches 35 on Antmaze, vs. 622 for Offline QDAC with the same backbone (App. E.7).
  • Combining goal-conditioned RL with fitness (GCIQL + fitness) remains below Offline QDAC on both tasks where it applies (527 vs. 622, 211 vs. 261; see W3).
  • Giving online QD methods the same data does not close the gap (see W2).

W2 (offline vs. online comparison). The locomotion datasets (RND, Replay, Expert) were generated to mirror the standard offline RL quality levels (random, replay, expert), in order to study how dataset quality affects each method (Section 5.2, Appendix E.4). They are an analysis tool, not evidence of interaction efficiency, and we make no claim that offline QD outperforms ME there. In contrast, our hard-exploration results use public OGBench datasets, collected independently of our work and without any QD algorithm.

Concerning the cost, data collection is a one-time cost, shared across every method and task that reuses the data. This is the premise of offline RL, where methods are not charged for the cost of collecting D4RL or OGBench datasets. Online QD, by contrast, pays the full interaction cost again for every new descriptor space or fitness function.

As requested by the reviewer, we add online QD baselines that use the same dataset D\mathcal{D} under the same online budget as ME: PGA-ME, DCG-ME and online QDAC with their replay buffers pre-filled with D\mathcal{D}; ME initialized from a behavior-conditioned BC policy trained on D\mathcal{D}. Results are in the table below (App. E.2, Table 12 of the revised version).

Table R1: QD score (mean ± std over 5 seeds) of online QD methods with access to D\mathcal{D}, under the same online budget as ME in Table 1. Table 12 in the revised paper.

MethodUse of D\mathcal{D}Antmaze ExploreAntmaze StitchAntmaze NavigateCube NoisyCube Play
ME (dataset-independent)None26±426±426±415±315±3
PGA-ME + D\mathcal{D}Replay buffer pre-filled with D\mathcal{D}36±332±434±624±424±3
DCG-ME + D\mathcal{D}Replay buffer pre-filled with D\mathcal{D}32±541±1233±618±216±2
QDAC + D\mathcal{D}Replay buffer pre-filled with D\mathcal{D}106±12112±15133±1663±971±10
ME init. from conditional BCBC on D\mathcal{D} distilled into initial population512±43536±36516±51102±13108±13
BD-DT (ours)Offline only482±48778±132448±5975±15112±13
Offline QDAC (ours)Offline only532±58696±136638±102226±42297±39

Pre-filling the replay buffer with D\mathcal{D} barely improves PGA-ME and DCG-ME over their from-scratch results (Table 11): their critics learn fitness, while the bottleneck in these tasks is discovering new regions of the descriptor space. Online QDAC benefits more (from 12 to 133 on Antmaze Navigate), as its SF critic learns compliance from D\mathcal{D}, but it remains far below Offline QDAC (638). In contrast, using D\mathcal{D} to learn a descriptor-conditioned policy offline, even by simple BC, yields a strong initial archive, which supports the premise of Offline QD. Without any interaction, Offline QDAC exceeds BC followed by online search in all settings.

We would welcome any further method the reviewer has in mind and will be glad to include it in our study.

AuthorsOpenReview ↗

Response to Reviewer 8vfy (2/3): offline baselines, generality

W3 (offline baselines). Several baselines from the families mentioned by the reviewer were already in the submission: offline GCRL (GCIQL, App. E.3), reward-conditioned imitation (DT, App. E.8), behavior-conditioned imitation (behavior-conditioned BC and QDT, App. E.7-E.8), and reward-aware, behavior-conditioned offline RL (IQL with a combined fitness and compliance reward, App. E.7). Since they were presented in separate appendices, we agree they were easy to miss, and in the revised version we gather them in a single table (App. E.3, Table 14), same as below.

To our knowledge, no existing offline method jointly optimizes fitness and compliance with a predefined descriptor space. Offline multi-behavior methods (SORL, DiveOff) learn behaviors in a latent space without a predefined descriptor, so they cannot be queried for a target zz without rollouts (App. B.2, Table 3). As no specific method is named in the review, we built reward-aware, behavior-conditioned baselines from existing paradigms:

  • GCIQL + fitness: GCIQL with an energy cost added to the goal-reaching reward (α\alpha selected by sweep); applicable to final-state descriptors only.
  • Filtered conditional BC: behavior-conditioned BC trained on the top-10% fitness trajectories of each descriptor.

The results are given in Table R2.

Table R2: QD score of offline baselines, averaged over datasets per environment (mean ± std, 5 seeds). Table 14 in the revised paper.

IngredientsMethodHalfcheetahWalkerAntmazeCube
Compliance onlyGCIQL (E.3)--466±46126±16
Behavior-conditioned BC (E.7)464±2341±5489±5194±13
QDT (E.8)450±2743±7297±3849±7
Compliance + fitnessIQL + combined reward (E.7)95±1643±1035±7130±34
GCIQL + fitness (new)--527±24211±15
Filtered conditional BC (new)366±2521±7261±5183±15
OursBD-DT502±5361±11569±5794±11
OursOffline QDAC1701±4886±21622±58261±36

Offline QDAC outperforms all baselines in all environments. The strongest baseline, GCIQL + fitness, remains below it on Antmaze and Cube: optimizing a single weighted reward trades compliance against fitness, whereas Offline QDAC's reward gate enforces compliance first. GCIQL has better compliance but lower QD score, since it ignores fitness. Filtered BC underperforms plain BC, as filtering leaves most descriptors with very few trajectories.

If the reviewer has any specific additional method in mind, we would be glad to include it in our study.


W4 (generality and failure cases). We agree with this assessment, and these limitations are the ones we report in Section 6. Indeed, our claim is about feasibility: as a first study, we aim to show that Offline QD is tractable, not to propose a single general-purpose method. Moreover, each limitation is predictable from the paradigm the method builds on, and in every environment at least one of the three methods recovers a substantial archive (Table 1). For instance, JEPA-ME fails on long-horizon, contact-rich dynamics, and BD-DT depends on dataset quality. We report these failures deliberately, since they indicate which approach suits which setting.

Regarding scope, our descriptor spaces go up to 4D (Ant_uni). We agree that real-world data and higher-dimensional descriptors are important next steps, and we state this explicitly in Section 6.

AuthorsOpenReview ↗

Response to Reviewer 8vfy (3/3): theory vs. practice, questions

W5 (theory vs. practice). On the dataset distribution. The reviewer is right that replacing the policy's state distribution with the dataset distribution requires a coverage assumption, which we explicitly formalize in Corollary 1: the bound transfers to dataset states at the cost of a concentrability factor CzC_z, which is small when the dataset covers the states the policy visits.

On the infinite horizon. Our analysis follows the infinite-horizon discounted setting that is standard in RL theory [1, 2, 3, 4], even though RL methods are evaluated on finite-horizon tasks. In the revised version, we state this scope explicitly (App. A.1) and add a finite-horizon bound that holds for the learned successor features (App. A.2, Proposition 2). For a horizon TT, let BDT=E[1T∑t<Tϕ(st)]\mathrm{BD}_T=\mathbb{E}\big[\frac1T\sum_{t<T}\phi(s_t)\big], let ρˉT,z\bar\rho_{T,z} be the average state distribution of πθ(⋅∣z)\pi_\theta(\cdot|z) over the episode, let ε\varepsilon be the Bellman residual of the learned SF (zero for the exact SF), and let $e(s,a,z)=\Vert (1-\gamma)\psi(s,a,z)-z\Vert $. Then, for every TT,

∥BDT−z∥≤EρˉT,z[e]⏟compliance+EρˉT,z∥ε∥⏟SF error+γ(1−γ)TE[e0+eT]⏟boundary.\Vert \mathrm{BD}_T-z\Vert \le\underbrace{\mathbb{E}_{\bar\rho_{T,z}}[e]}_{\text{compliance}}+\underbrace{\mathbb{E}_{\bar\rho_{T,z}}\Vert \varepsilon\Vert }_{\text{SF error}}+\underbrace{\frac{\gamma}{(1-\gamma)T} \mathbb{E}[e_0+e_T]}_{\text{boundary}}.

Proposition 1 is its limit for exact SF and T→∞T\to\infty. The proof, and a version on dataset states with a coverage coefficient (Corollary 2), are in App. A.2. Concerning the final-state descriptors, we agree that the guarantee for Offline QDAC covers additive descriptors only, as explicitly stated in Section 6.

On the empirical validity of App. E.11 (E.9 in the original submission). As an upper bound, Proposition 1 is expected to be loose. What matters is that it holds. In App. E.11, the surrogate upper-bounds the true error for 93% of queried descriptors on Halfcheetah Expert despite the dataset substitution, and the remaining 7% measure the cost of that substitution rather than violations of Proposition 1, which holds under the on-policy distribution.

[1] J. Schulman, S. Levine, P. Abbeel, M. Jordan, P. Moritz. Trust Region Policy Optimization. ICML 2015 (Section 2).

[2] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018 (Section 3.1).

[3] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. van Hasselt, D. Silver. Successor Features for Transfer in Reinforcement Learning. NeurIPS 2017 (Section 2).

[4] L. Grillotti, M. Faldor, B. G. León, A. Cully. Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics. ICML 2024 (Section 4.1).


Q1 (dataset collection cost). See our answer to W2.


Q2 (offline baselines). See our answer to W3.


Q3 (algorithmic contribution). See our answer to W1.


Q4 (dataset coverage). In theory, Corollary 1 makes this dependence explicit: compliance degrades with the concentrability factor CzC_z, i.e. as the policy visits states the dataset does not cover. Empirically, three experiments already in the paper address this question:

  • Out-of-support descriptors (App. E.5). When training only on trajectories from one contiguous part of the descriptor space, Offline QDAC degrades gradually, unlike BD-DT, which fails sharply as queried descriptors move away from the covered region (Figure 7).
  • Data far from the learned policies (Table 1). The RND datasets are collected by an exploration policy, whose states differ from those visited by high-fitness policies. Offline QDAC remains the strongest offline method on them (e.g. 231 vs. 86 for BD-DT on Halfcheetah).
  • Limited diversity (App. E.4). We mix diverse data with fitness-maximizing data. Offline QDAC needs less diverse data than BD-DT to saturate coverage.

Nonetheless, performance depends on dataset quality (e.g. 2461 on Halfcheetah Expert vs. 231 on RND). This dependence is inherent to offline learning: offline RL methods are known to vary with dataset quality, which is why benchmarks such as D4RL evaluate them across quality levels, as we do.

Reviewer_SSf6OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Offline Quality-Diversity: learning a repertoire of high-fitness policies across a predefined behavioral descriptor space from a fixed dataset. It explores three approaches—Offline QDAC, JEPA-ME, and BD-DT—using successor features, learned world models, and conditional sequence modeling, respectively. The results demonstrate the feasibility of offline repertoire learning and its usefulness for initializing subsequent online search.

Critical strengths
  1. The problem formulation is meaningful: extracting diverse, useful behaviors from existing data extends offline learning beyond finding a single reward-maximizing policy.
  2. The three approaches provide an informative comparison of different ways to address offline behavioral compliance. Their complementary strengths and failure cases offer useful insights.
  3. Experiments across six environments and fifteen datasets provide substantial evidence for the central feasibility claim. The component analyses and improvements from offline initialization further support the contribution.
Critical weaknesses
  1. Offline QDAC’s successor-feature objective does not generally preserve the behavioral constraint in the problem formulation. For terminal-state descriptors, matching discounted average features differs from reaching a specified endpoint and may favor different policies, even with perfect predictions. Although acknowledged, this limits the strongest method’s applicability. A descriptor-specific formulation would better align optimization with the intended constraint.
  2. Offline QDAC’s treatment of offline extrapolation remains incomplete. Its actor directly optimizes critic and successor-feature predictions at policy-generated actions; IQL-style critic updates alone do not constrain these actions to dataset support. Errors can therefore affect both fitness optimization and the behavioral-compliance gate. The empirical results demonstrate usefulness, but stronger control of action support or prediction uncertainty would make this mechanism more reliable.
  3. The online-initialization experiments demonstrate the benefit of starting from an offline archive, but do not isolate the contribution of the proposed learning objectives. It remains unclear whether similar gains could be obtained by simply imitating the dataset’s diverse behaviors. Using the existing behavior-conditioned BC baseline from Appendix E.5 to initialize MAP-Elites, with the same online budget on one representative task, would provide a useful control.
Questions for authors

Please answer the points in weakness.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SSf6 (1/3): final-state descriptors

We thank the reviewer for the constructive review, and for finding the formulation meaningful, the comparison of the three approaches informative, and the experiments substantial evidence for our feasibility claim. In response, we (i) make precise when the SF objective matches final-state descriptors, and how it could target them directly (W1), (ii) clarify how OOD actions are handled and add ablations of the BC term and of alternatives that avoid policy-action queries (W2, Tables R1-R2), and (iii) run the suggested BC-initialized control on all five hard-exploration datasets, extended to QDT (W3, Table R3). All numbers refer to the revised version; changes are shown in blue.


W1 (final-state descriptors). We agree. The mismatch can be stated precisely: the SF objective constrains the discounted average of future features:

(1−γ)ψ(s,a,z)=EH∼Geom(1−γ)[ϕ(st+H)],(1-\gamma) \psi(s,a,z) = \mathbb{E}_{H\sim\mathrm{Geom}(1-\gamma)}\big[\phi(s_{t+H})\big],

i.e. a weighted average of the features over the next ≈100 effective steps (for γ=0.99\gamma=0.99), whereas a final-state descriptor constrains the feature at the final step only. The two coincide for policies that reach zz and remain there: once zz is reached, both the discounted average and the final feature equal zz. This explains why Offline QDAC performs well on Antmaze and Cube (the ant/robotic arm can stop), and is the strongest method on 4 of their 5 datasets (Table 1).

Following the reviewer's suggestion, we believe that successor features can theoretically be extended to constrain the feature at the exact final step, by indexing the successor features with the remaining time kk:

Ψ(s,a,k,z)=Eπz[ϕ(st+k)∣st=s,at=a],Ψ(s,a,k,z)=E[Ψ(s′,a′,k−1,z)],Ψ(s,a,1,z)=E[ϕ(s′)].\Psi(s,a,k,z) = \mathbb{E}_{\pi_z}\big[\phi(s_{t+k}) \mid s_t = s, a_t = a\big], \qquad \Psi(s,a,k,z) = \mathbb{E}\big[\Psi(s',a',k-1,z)\big], \quad \Psi(s,a,1,z) = \mathbb{E}\big[\phi(s')\big].

Ψ\Psi predicts the final descriptor directly, so our gate applies unchanged (∥Ψ−z∥<δ\Vert \Psi - z\Vert < \delta, in descriptor units). This requires the policy to also take the remaining time as input (time-awareness [1]). We leave this idea to future work.

[1] F. Pardo, A. Tavakoli, V. Levdik, P. Kormushev. Time Limits in Reinforcement Learning. ICML 2018.

AuthorsOpenReview ↗

Response to Reviewer SSf6 (2/3): OOD actions

W2 (OOD actions). The reviewer is right that IQL-style critic updates do not, by themselves, keep the actor within the dataset support: the actor and the SF targets both query actions generated by the policy. Several mechanisms limit extrapolation in practice:

  • Critic: QQ and VV are trained in-sample with IQL [2], so their targets never use out-of-distribution actions.
  • Gate: the reward gradient is only active where the policy is already compliant, which restricts Q-maximization to those states only.
  • Successor features: features are bounded, so ψ\psi is bounded by ∥ϕ∥/(1−γ)\Vert \phi\Vert /(1-\gamma), and the SF update is a policy evaluation without a max operator, which avoids the overestimation feedback loop of Q-learning.
  • Behavioral constraint: on Antmaze and Cube, we follow the standard OGBench practice [3] of regularizing the actor with a BC term (DDPG+BC [4]), with coefficient 0.3 (Appendix D.1.2).

Effect of the BC term. Constraining actions toward the dataset helps substantially on the OGBench tasks but hurts on locomotion (Table R1). We therefore apply it only on Antmaze and Cube, in line with common practice on this benchmark [3, 4] (Appendix D.1.2).

Table R1: Effect of the BC term on Offline QDAC, averaged over datasets per environment (mean ± std, 5 seeds). Table 16 in the revised paper.

VariantHalfcheetahWalkerAntmazeCube
Without BC1701±4886±21287±27155±21
With BC (α\alpha = 0.3)1243±2652±9622±58261±36

Alternatives to our design. We also compare three variants, each replacing one component of Offline QDAC (App. E.6, Table 15 of the revised version):

  • TD3 critic: removes the conservative in-sample critic.
  • AWR actor: restricts policy extraction to dataset actions, weighted by advantage.
  • ψ\psi-IQL: learns the successor features in-sample, mirroring IQL. We learn an action-independent function E(s,z)E(s,z) by expectile regression on dataset actions, weighting more heavily those whose successor features are closer to the target:

LE=E(s,a)∼D,z[∣τ−1[u<0]∣∥ψ(s,a,z)−E(s,z)∥22],u=∥(1−γ)E(s,z)−z∥−∥(1−γ)ψ(s,a,z)−z∥,\mathcal{L}_E = \mathbb{E}_{(s,a)\sim\mathcal{D}, z}\Big[ \big|\tau - \mathbb{1}[u < 0]\big| \big\Vert \psi(s,a,z) - E(s,z)\big\Vert _2^2\Big], \qquad u = \big\Vert (1-\gamma)E(s,z) - z\big\Vert - \big\Vert (1-\gamma)\psi(s,a,z) - z\big\Vert ,

and bootstrap the successor features through it, ψ(s,a,z)←ϕ(s)+γE(s′,z)\psi(s,a,z) \leftarrow \phi(s)+\gamma E(s',z), so that no policy action is queried. As τ→1\tau\to1, E(s,z)E(s,z) approaches the successor features of the most compliant dataset action, ψ(s,a⋆,z)\psi(s,a^\star,z) with $a^\star = \arg\min_{a \in \mathrm{supp} \mathcal{D}(\cdot|s)} \Vert (1-\gamma)\psi(s,a,z) - z\Vert $.

Table R2: Offline QDAC with alternative policy extraction and SF learning, averaged over datasets per environment (mean ± std, 5 seeds). Table 15 in the revised paper.

CriticActorSF learningHalfcheetahWalkerAntmazeCube
TD3 (no conservatism)Q-maximizationTD with policy actions1252±5484±24523±65226±31
IQLAWRTD with policy actions1422±3473±16564±61209±22
IQLQ-maximizationψ\psi-IQL (in-sample expectile)1197±4271±18514±35192±32
IQLQ-maximizationTD with policy actions (ours)1701±4886±21622±58261±36

Our design performs best in all four environments. Removing the conservative critic (TD3) lowers performance, except on Walker, where all variants overlap within one standard deviation. AWR underperforms our gradient-based extraction in every environment, matching the findings of Park et al. [5]. ψ\psi-IQL is the weakest option. We believe this is because IQL estimates the value of the best behavior in the data, which is what QQ should represent, whereas successor features must describe the current policy, since they measure its own compliance. ψ\psi-IQL instead estimates the successor features of the most compliant dataset behavior, which differs from the current policy's behavior. We include it in App. E.6 of the revised version as a negative result, with this discussion.

Finally, the disagreement between the two heads of our SF network could provide an uncertainty estimate to penalize the actor. We mention this direction in Section 6 of the revised version, and leave it for future work.

[2] I. Kostrikov, A. Nair, S. Levine. Offline Reinforcement Learning with Implicit Q-Learning. ICLR 2022.

[3] S. Park, K. Frans, B. Eysenbach, S. Levine. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.

[4] S. Fujimoto, S. S. Gu. A Minimalist Approach to Offline Reinforcement Learning. NeurIPS 2021.

[5] S. Park, K. Frans, S. Levine, A. Kumar. Is Value Learning Really the Main Bottleneck in Offline RL? NeurIPS 2024.

AuthorsOpenReview ↗

Response to Reviewer SSf6 (3/3): BC-initialized control

W3 (BC-initialized control). We ran this control on all five hard-exploration datasets rather than one, and extended it to QDT, a second imitation baseline. Each offline policy is distilled into an initial population for ME, with the same protocol and online budget as Section 5.3. We focus on Antmaze and Cube because that is where the initialization matters most. In the revised version, the results are in App. E.12 (Table 19) and discussed in Section 5.3.

Table R3: QD score after online search, by initialization (mean ± std, 5 seeds). Table 19 in the revised paper.

InitializationAntmaze ExploreAntmaze StitchAntmaze NavigateCube NoisyCube Play
None (ME from scratch)26±426±426±415±315±3
Behavior-conditioned BC512±43536±36516±51102±13108±13
QDT492±45630±60437±2298±12112±22
BD-DT (ours)546±53825±70517±4983±6124±9
Offline QDAC (ours)605±31797±63734±61257±22337±29

This shows that imitating the dataset alone already gives a much better initialization than starting from scratch. However, Offline QDAC initialization outperforms both imitation baselines in all five settings, by up to 3.1x on Cube, so its learning objective contributes beyond imitation.

AuthorsOpenReview ↗

Follow-up on W1: time-aware compliance critic implemented

In our answer to W1, we proposed indexing the compliance critic by the remaining time, and left it to future work. We have now implemented it (see our global comment). The critic predicts the expected squared distance of a recursive descriptor statistic rather than the expected descriptor. This also covers the challenging range descriptor from Pointmass. It improves Offline QDAC on Pointmass range (487 → 1896), Antmaze Explore (+60%) and Cube Play (+36%). We thank the reviewer for this suggestion.

Reviewer_gPgxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Offline Quality-Diversity (Offline QD), where the objective is to recover a repertoire of high-performing policies covering a predefined behavioral descriptor space from a static dataset, without environment interaction during training. It explores three approaches: Offline QDAC, which uses successor features to encourage behavioral compliance; JEPA-ME, which performs quality-diversity search in a learned world model; and BD-DT, which jointly conditions a decision transformer on behavioral descriptors and returns. Experiments cover six environments and multiple dataset regimes, including additional analyses of behavioral compliance, dataset coverage, component ablations, and subsequent online improvement.

Critical strengths
  1. The problem is clearly motivated. Recovering diverse, controllable behaviors from existing data is practically relevant when further interaction is expensive or unavailable. The formulation explicitly distinguishes behavioral compliance from fitness maximization.

  2. The three approaches provide a useful initial exploration of different ways to address the absence of online behavioral evaluation. Their contrasting performance helps identify relevant challenges, including descriptor structure, model rollout error, and dependence on dataset quality.

  3. The experimental coverage is substantial. Beyond the main comparisons, the appendix includes stronger online QD baselines, a goal-conditioned offline RL baseline, component ablations, dataset-quality studies, and analyses of generalization beyond observed descriptors. The authors also acknowledge important limitations and failure cases.

Critical weaknesses

1. The advantage over existing offline methods remains unclear. GCIQL achieves higher compliance at the evaluation threshold on Antmaze and Cube, but its fitness and QD score are omitted. These comparisons are needed to establish the benefit of jointly optimizing fitness and diversity.

2. Offline QDAC’s handling of out-of-distribution actions is insufficiently justified. Both the actor objective and SF updates query current-policy actions, so IQL-style critic training alone does not preserve standard IQL’s avoidance of unseen-action queries.

3. The theory does not directly cover the implemented algorithm. The proofs assume exact SF and infinite-horizon stationary descriptors, whereas experiments use learned SF and finite episodes. Neither SF estimation error nor finite-horizon discrepancies are accounted for.

Questions for authors
  1. Could you report fitness, coverage, and QD score for GCIQL on Antmaze and Cube? Would the proposed methods retain an advantage over a goal-conditioned baseline that also incorporates the energy-efficiency objective?

  2. What mechanism controls out-of-distribution action exploitation in the actor objective and SF updates? Please clarify the complete policy objective and compare the current approach with advantage-weighted policy extraction or an explicit behavioral constraint.

  3. Can the compliance analysis be extended to finite-horizon descriptors and approximate SF, including estimation-error and coverage terms? Alternatively, could the theoretical claims be narrowed and the surrogate evaluated on lower-coverage datasets and unsupported target descriptors?

  4. Are policies reassigned to cells according to their achieved descriptors, with duplicate occupancy removed, when calculating QD score? Do the conclusions hold under this evaluation and when online search is initialized with BC/QDT policies trained on the same data?

7 comments
AuthorsOpenReview ↗

Response to Reviewer gPgx (1/4): GCIQL with fitness

We thank the reviewer for the careful review, and for finding the problem clearly motivated, the three approaches a useful initial exploration, and the experimental coverage substantial. In response, we (i) report the full QD metrics of GCIQL and add GCIQL + fitness (Q1), (ii) state the complete objective and compare alternative policy extraction and SF learning (Q2), (iii) add a finite-horizon bound for learned SF (Q3), and (iv) report the reassigned QD score and online search initialized from BC and QDT (Q4). W1, W2 and W3 are addressed in Q1, Q2 and Q3. All numbers refer to the revised version; changes are shown in blue.


Q1 (GCIQL with fitness). In the revised version (App. E.3, Table 13), we report coverage, fitness and QD score for GCIQL, and add a goal-conditioned baseline that also incorporates fitness: GCIQL + fitness, which uses the reward r=1[∥ϕ(s′)−g∥<ε]−1−α∥a∥2/dAr = \mathbb{1}[\Vert \phi(s') - g\Vert < \varepsilon] - 1 - \alpha \Vert a\Vert _2/\sqrt{d_A}, i.e. the goal-reaching reward with an energy cost (α\alpha selected by sweep in $\lbrace 0.1, 0.3, 1.0\rbrace $). Both use the same architecture and IQL hyperparameters as Offline QDAC.

Table R1: Coverage (% of compliant skills), mean fitness of compliant skills, and QD score on the final-state tasks, averaged over datasets (mean ± std, 5 seeds). Table 13 in the revised paper.

MethodAntmaze CoverageAntmaze FitnessAntmaze QD scoreCube CoverageCube FitnessCube QD score
GCIQL48.160.39±0.03466±4643.240.12±0.01126±16
GCIQL + fitness28.320.74±0.04527±2424.240.35±0.03211±15
Offline QDAC (ours)31.080.80±0.06622±5812.400.84±0.06261±36

GCIQL reaches the highest coverage, but low fitness, since it optimizes goal-reaching only. Adding the energy objective raises fitness at the cost of coverage: a single weighted reward trades the two off. Offline QDAC, which gates the reward gradient on compliance, retains a clear advantage, achieving the highest QD score on both tasks.

AuthorsOpenReview ↗

Response to Reviewer gPgx (2/4): OOD actions

Q2 (OOD actions). The reviewer is right: IQL's in-sample critic training avoids unseen actions in the critic targets, but our actor objective and SF targets do query current-policy actions. In the revised version, we correct the corresponding statement in Appendix B.2, discuss out-of-distribution actions in Section 4.1, and state the BC term in Appendix D.1.2. The complete objective is

J(πθ)=Es∼D,z∼U(Z)[−1[ℓ(s,z)<δ]Ea∼πθ[Q(s,a,z)]+ℓ(s,z)]+αBCLBCJ(\pi_\theta) = \mathbb{E}_{s\sim\mathcal{D}, z\sim\mathcal{U}(\mathcal{Z})}\Big[-\mathbb{1}[\ell(s,z)<\delta] \mathbb{E}_{a\sim\pi_\theta}[Q(s,a,z)] + \ell(s,z)\Big] + \alpha_{\text{BC}} \mathcal{L}_{\text{BC}}

with ℓ(s,z)=Ea∼πθ∥(1−γ)ψ(s,a,z)−z∥2\ell(s,z) = \mathbb{E}_{a\sim\pi_\theta}\Vert (1-\gamma)\psi(s,a,z)-z\Vert _2 and LBC=E(s,a)∼D[∥πθ(s,z)−a∥22]\mathcal{L}_{\text{BC}} = \mathbb{E}_{(s,a)\sim\mathcal{D}}\big[\Vert \pi_\theta(s,z)-a\Vert _2^2\big], where αBC=0.3\alpha_{\text{BC}}=0.3 on Antmaze and Cube and 00 otherwise.

Mechanisms limiting OOD exploitation:

  • Critic: QQ and VV are trained in-sample with IQL, so their targets never use out-of-distribution actions.
  • Gate: the reward gradient is only active where the policy is already compliant (ℓ<δ\ell < \delta), which restricts Q-maximization to those states.
  • Behavioral constraint: on Antmaze and Cube, we follow the standard OGBench practice of regularizing policy extraction with a BC term [1, 2].
  • SF: features are bounded, so ψ\psi is bounded by ∥ϕ∥/(1−γ)\Vert \phi\Vert /(1-\gamma), and the SF update is a policy evaluation without a max operator, which avoids the overestimation feedback loop of Q-learning.

Comparison with alternative policy extraction and SF learning, as requested (App. E.6, Table 15 of the revised version):

Table R2: QD score of Offline QDAC variants, averaged over datasets per environment (mean ± std, 5 seeds). Table 15 in the revised paper.

CriticActorSF learningHalfcheetahWalkerAntmazeCube
TD3 (no conservatism)Q-maximizationTD with policy actions1252±5484±24523±65226±31
IQLAWRTD with policy actions1422±3473±16564±61209±22
IQLQ-maximizationψ\psi-IQL (in-sample expectile)1197±4271±18514±35192±32
IQLQ-maximizationTD with policy actions (ours)1701±4886±21622±58261±36

Our variant performs best in all four environments. Replacing Q-maximization with AWR lowers performance everywhere, consistent with Park et al. [3], who find that gradient-based policy extraction generally makes better use of a learned value function than weighted regression. Removing conservatism (TD3 instead of IQL) also lowers performance, except on Walker, where all variants are within one standard deviation.

In addition, we tried learning the successor features in-sample, mirroring IQL. We learn an action-independent function E(s,z)E(s,z) by expectile regression on dataset actions, weighting more heavily those whose successor features are closer to the target:

LE=E(s,a)∼D,z[∣τ−1[u<0]∣∥ψ(s,a,z)−E(s,z)∥22],u=∥(1−γ)E(s,z)−z∥−∥(1−γ)ψ(s,a,z)−z∥,\mathcal{L}_E = \mathbb{E}_{(s,a)\sim\mathcal{D}, z}\Big[ \big|\tau - \mathbb{1}[u < 0]\big| \big\Vert \psi(s,a,z) - E(s,z)\big\Vert _2^2\Big], \qquad u = \big\Vert (1-\gamma)E(s,z) - z\big\Vert - \big\Vert (1-\gamma)\psi(s,a,z) - z\big\Vert ,

and bootstrap the successor features through it, ψ(s,a,z)←ϕ(s)+γE(s′,z)\psi(s,a,z) \leftarrow \phi(s)+\gamma E(s',z), so that no policy action is queried. As τ→1\tau\to1, E(s,z)E(s,z) approaches the successor features of the most compliant dataset action, ψ(s,a⋆,z)\psi(s,a^\star,z) with $a^\star = \arg\min_{a \in \mathrm{supp} \mathcal{D}(\cdot|s)} \Vert (1-\gamma)\psi(s,a,z) - z\Vert $. This performs worse than our method. We believe this is because IQL estimates the value of the best behavior in the data, which is what QQ should represent, whereas successor features must describe the current policy, since they measure its own compliance. ψ\psi-IQL instead estimates the successor features of the most compliant dataset behavior, which differs from the current policy's behavior. We include it in App. E.6 of the revised version as a negative result, with this discussion.

AuthorsOpenReview ↗

Response to Reviewer gPgx (3/4): BC term, finite-horizon theory

Effect of the explicit behavioral constraint (App. E.6, Table 16 of the revised version):

Table R3: QD score of Offline QDAC with and without the BC term, averaged over datasets per environment (mean ± std, 5 seeds). Table 16 in the revised paper.

VariantHalfcheetahWalkerAntmazeCube
Without BC1701±4886±21287±27155±21
With BC (α\alpha = 0.3)1243±2652±9622±58261±36

The BC term improves performance on Antmaze and Cube but reduces it on Halfcheetah and Walker. We therefore follow the OGBench convention and use it only on those tasks.

[1] S. Fujimoto, S. S. Gu. A Minimalist Approach to Offline Reinforcement Learning. NeurIPS 2021.

[2] S. Park, K. Frans, B. Eysenbach, S. Levine. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.

[3] S. Park, K. Frans, S. Levine, A. Kumar. Is Value Learning Really the Main Bottleneck in Offline RL? NeurIPS 2024.


Q3 (finite-horizon theory). Proposition 1 indeed assumes exact SF, with an infinite horizon, as is standard in RL theory [4, 5, 6, 7]. Nevertheless, in the revised version, we added a finite-horizon bound that holds for the learned SF, with explicit estimation-error and coverage terms (Proposition 2 and Corollary 2, Appendix A.2):

Finite horizon and learned SF. For a horizon TT, let BDT=E[1T∑t<Tϕ(st)]\mathrm{BD}_T=\mathbb{E}\big[\frac1T\sum_{t<T}\phi(s_t)\big], let ρˉT,z\bar\rho_{T,z} be the average state distribution of πθ(⋅∣z)\pi_\theta(\cdot|z) over the episode, let ε(s,a,z)=ψ(s,a,z)−ϕ(s)−γEs′,a′∼πθ[ψ(s′,a′,z)]\varepsilon(s,a,z)=\psi(s,a,z)-\phi(s)-\gamma \mathbb{E}_{s', a'\sim\pi_\theta}[\psi(s',a',z)] be the Bellman residual of the learned SF (zero for the exact SF), and let $e(s,a,z)=\Vert (1-\gamma)\psi(s,a,z)-z\Vert $. Proposition 2 states that, for every TT,

∥BDT−z∥≤EρˉT,z[e]⏟compliance+EρˉT,z∥ε∥⏟SF error+γ(1−γ)TE[e0+eT]⏟boundary.\Vert \mathrm{BD}_T-z\Vert \le\underbrace{\mathbb{E}_{\bar\rho_{T,z}}[e]}_{\text{compliance}}+\underbrace{\mathbb{E}_{\bar\rho_{T,z}}\Vert \varepsilon\Vert }_{\text{SF error}}+\underbrace{\frac{\gamma}{(1-\gamma)T} \mathbb{E}[e_0+e_T]}_{\text{boundary}}.

The proof is a telescoping sum and needs no stationarity assumption. Proposition 1 is its limit for exact SF (ε=0\varepsilon=0) and T→∞T\to\infty.

Coverage. Corollary 2 moves the first two terms to dataset states, with the concentrability coefficient CT,z=sup⁡sρˉT,z(s)/dD(s)C_{T,z}=\sup_s\bar\rho_{T,z}(s)/d_\mathcal{D}(s):

∥BDT−z∥≤CT,z(Es∼D[ℓ(s,z)]+Es∼D,a∼πθ∥ε∥)+γ(1−γ)TE[e0+eT],\Vert \mathrm{BD}_T-z\Vert \le C_{T,z}\Big(\mathbb{E}_{s\sim\mathcal{D}}[\ell(s,z)]+\mathbb{E}_{s\sim\mathcal{D}, a\sim\pi_\theta}\Vert \varepsilon\Vert \Big)+\frac{\gamma}{(1-\gamma)T} \mathbb{E}[e_0+e_T],

where ℓ\ell is the compliance loss of Eq. 7.

This bound shows which terms the training controls: the SF TD loss reduces the SF error at dataset actions, and the actor minimizes the compliance term on dataset states. The coverage coefficient CT,zC_{T,z} and the SF error at actions outside the data cannot be estimated offline, as is standard for concentrability coefficients in offline RL.

Narrowed claims. The analysis covers additive descriptors over fixed-length episodes. For final-state descriptors (Antmaze, Cube, Pointmass last state) and non-additive ones (Pointmass range), our results are empirical, and we now state this in Section 6. As suggested, we also evaluated the surrogate on lower-coverage data and unsupported targets: the percentage of error rises from 7% on Halfcheetah Expert to 21% on Halfcheetah RND and 42% on a truncated dataset, consistent with Corollary 2.

[4] J. Schulman, S. Levine, P. Abbeel, M. Jordan, P. Moritz. Trust Region Policy Optimization. ICML 2015 (Section 2).

[5] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. ICML 2018 (Section 3.1).

[6] A. Barreto, W. Dabney, R. Munos, J. J. Hunt, T. Schaul, H. van Hasselt, D. Silver. Successor Features for Transfer in Reinforcement Learning. NeurIPS 2017 (Section 2).

[7] L. Grillotti, M. Faldor, B. G. León, A. Cully. Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics. ICML 2024 (Section 4.1).

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of the rebuttal

We thank the reviewers and the AC for their efforts. We addressed the concerns with new experiments, a new finite-horizon bound, and corrections to the paper, summarized in the table below, with changes in blue in the revised version of the paper.

ConcernRaised byWhat we didRevised version
OOD actions8vfy, gPgx, SSf6, R5DpCorrected our claim (actor and SF targets do query policy actions), stated the complete objective with the BC term, compared AWR, in-sample SF (ψ\psi-IQL), TD3 and the BC termSec. 4.1, App. B.2, D.1.2, E.6 (Tables 15-16)
Online QD with the same data8vfyPGA-ME, DCG-ME, online QDAC with replay buffers pre-filled with D\mathcal{D}; ME initialized from BCApp. E.2 (Table 12)
Offline baselines with fitness8vfy, gPgxFull QD metrics for GCIQL; new GCIQL + fitness and filtered conditional BC; all baselines in one tableApp. E.3 (Tables 13-14)
BC-initialized online searchSSf6, gPgxME initialized from BC and QDTSec. 5.3, App. E.12 (Table 19)
Theory vs. practice8vfy, gPgx, SSf6New finite-horizon bound for the learned SF, with explicit SF-error and coverage termsApp. A.1, A.2 (Prop. 2, Cor. 2), Sec. 4.1, 6
QD score protocolgPgxQD score recomputed with reassignment to achieved cellsgPgx answer, Table R4
Novelty8vfy, R5DpContribution stated as the problem formulation with evidence of tractability; Offline-QD-specific challenges gathered in one sectionSec. 1, Sec. 6

Where the reviewers were right, we corrected the paper: the OOD statement (Sec. 4.1, App. B.2), the scope of the theory (App. A.1, Sec. 6) and the framing of our contribution (Sec. 1).

Main findings of the new experiments

  • The gains come from the learning objective, not only from data access.
    • Pre-filling the replay buffers with D\mathcal{D} barely helps PGA-ME and DCG-ME (at most 41 on Antmaze, vs. 26 for ME without data). Online QDAC benefits more but stays far below Offline QDAC (at most 133 vs. 532-696 on Antmaze).
    • Imitating D\mathcal{D} is more effective: ME initialized from BC reaches 512-536 on Antmaze.
    • Without any interaction, Offline QDAC exceeds BC followed by online search in all five settings (e.g. 638 vs. 516 on Antmaze Navigate, 297 vs. 108 on Cube Play).
    • Offline QDAC initialization also outperforms BC and QDT initialization in all five settings.
  • Offline QDAC outperforms all offline baselines in all four environments. The strongest one, GCIQL + fitness, reaches 527 vs. 622 on Antmaze and 211 vs. 261 on Cube.
  • OOD actions have a limited effect in practice. Our design performs best in all four environments against AWR, ψ\psi-IQL and a non-conservative TD3 critic. The BC term is essential on Antmaze and Cube (287 → 622 on Antmaze) but hurts on locomotion, so we use it only there, as is standard on OGBench.
  • Conclusions are robust to the QD score protocol. With reassignment, the ranking of offline methods is unchanged in all 16 settings.

Please note that new appendix sections shift the numbering of Appendix E (e.g. E.9 of the submission is now E.11). All pointers in our answers refer to the revised version.

AuthorsOpenReview ↗

New result: a time-aware compliance critic for final-state and other non-additive descriptors

Summary

Following Reviewer SSf6's suggestion (W1), which we had left to future work, we replaced the successor features (SF) in Offline QDAC with a time-aware compliance critic QcQ_c. It removes the failure of Offline QDAC on the Pointmass range descriptor (487 → 1896, comparable to JEPA-ME 1926 ± 273 and ME 2019 ± 202), clearly improves Antmaze Explore (+60%) and Cube Play (+36%), and comes with a finite-horizon bound that covers final-state descriptors (8vfy W4-W5, gPgx Q3). It is also a mechanism specific to Offline QD (8vfy W1, R5Dp W2). Table 1 and our earlier conclusions are unchanged.


Method

Motivation. As noted in our answer to SSf6 (W1), (1−γ)ψ(1-\gamma)\psi measures the feature at a geometric time about 1/(1−γ)=1001/(1-\gamma)=100 steps ahead, not at the end of the episode, and it cannot represent non-additive descriptors such as the range (max minus min of the positions). Offline and online QDAC both fail on Pointmass range (487 and 418, vs. 2019 for ME).

The new critic. Most descriptors used in QD can be computed recursively along a trajectory: m0=u0(s0)m_0=u_0(s_0), mt+1=u(mt,st+1)m_{t+1}=u(m_t,s_{t+1}) and BD(τ)=g(mT)\mathrm{BD}(\tau)=g(m_T). For a final-state descriptor, mt=ϕ(st)m_t=\phi(s_t); for the range, mtm_t is the running min and max; for a mean, the running mean. We condition the critic and the policy on the augmented state (s,m,k)(s,m,k), where k=T−tk=T-t is the number of steps left, which makes any such descriptor Markov. The critic Qc(s,m,k,a,z)Q_c(s,m,k,a,z) is trained by TD with the target

y=−∥g(m′)−z∥2 if k=1,y=−(1−γ)∥g(m′)−z∥2+γQˉc(s′,m′,k−1,a′,z) otherwise,y=-\Vert g(m')-z\Vert^2 \text{ if } k=1, \qquad y=-(1-\gamma)\Vert g(m')-z\Vert^2+\gamma\bar{Q}_c(s',m',k-1,a',z) \text{ otherwise},

with m′=u(m,s′)m'=u(m,s') and a′a' from the current policy. −Qc-Q_c is the expected squared distance to zz at the stopping time min⁡(H,k)\min(H,k), H∼Geom(1−γ)H\sim\mathrm{Geom}(1-\gamma); for γ=1\gamma=1, this is the end of the episode. Since kk and mm have known dynamics, both are relabeled for every dataset transition, as the descriptor zz already is, so only (s,a,s′)(s,a,s') comes from D\mathcal{D}. The fitness critic (IQL) and the form of Eq. 7 are unchanged: the policy additionally takes (m,k)(m,k) as input, and the SF term is replaced by −Qc\sqrt{-Q_c}. As for the SF, the QcQ_c targets query policy actions; we use a twin-critic minimum with target smoothing, and the same BC term on Antmaze and Cube.

Choice of γ\gamma. The exact critic (γ=1\gamma=1) trains on Pointmass (T=200T=200), but not on the 1000-step tasks, where the terminal signal must propagate through 1000 bootstrap steps. There we use γ=0.99\gamma=0.99, which gives a learning signal at every step.

Guarantee. For any distribution η\eta over kk (the one the actor's loss averages over), with dtπd^\pi_t the on-policy distribution of (s,m)(s,m) at time tt:

∥BD(π(⋅∣z))−z∥2≤Cγ,ηEk∼ηEdT−kπ,π[−Qcπ],Cγ,η=1Ek∼η[γk−1].\Vert \mathrm{BD}(\pi(\cdot|z))-z\Vert^2\le C_{\gamma,\eta}\mathbb{E}_{k\sim \eta}\mathbb{E}_{d^\pi_{T-k},\pi}\big[-Q_c^\pi\big], \qquad C_{\gamma,\eta}=\frac{1}{\mathbb{E}_{k\sim \eta}[\gamma^{k-1}]}.

Unlike Propositions 1 and 2, this covers any recursive descriptor, final-state descriptors included.


Results

(QD score, mean over 5 seeds, same protocol as Table 1):

SettingOffline QDAC (SF)Offline QDAC (QcQ_c)Best other method
Pointmass range487 ± 461896 ± 166ME 2019 ± 202
Pointmass last state2042 ± 3022194 ± 211ME 2321 ± 254
Antmaze Explore532 ± 58850 ± 65BD-DT 482 ± 48
Antmaze Navigate638 ± 102604 ± 120BD-DT 448 ± 59
Antmaze Stitch696 ± 136656 ± 137BD-DT 778 ± 132
Cube Noisy226 ± 42254 ± 47BD-DT 75 ± 15
Cube Play297 ± 39405 ± 51BD-DT 112 ± 13
  • On Pointmass range, the new critic removes the failure of Offline QDAC (487→1896, ×3.9) and is comparable to the best methods, offline (JEPA-ME 1926 ± 273) and online (ME 2019 ± 202).
  • On final-state descriptors, it clearly improves Antmaze Explore (+60%) and Cube Play (+36%). It is higher but within the standard deviation of the SF on Pointmass last state and Cube Noisy, and slightly lower but within it on Navigate and Stitch.

Closest work. Temporal difference models [1] also condition a critic on the remaining horizon, but for reaching a single goal state. QcQ_c differs in three ways: it applies to any recursively computable descriptor (final state, range, mean), not only to goal states; it is used as a compliance constraint jointly with a fitness objective; and it is learned fully offline, with the descriptor statistic mm and the remaining time kk relabeled counterfactually on every transition.

Revised version. The method and results are in App. E.13 and the proof in App. A.3, with one paragraph in Section 4.1 and the limitation in Section 6 updated. We keep Table 1 unchanged and present the new critic as an extension of Offline QDAC. Additions made during the discussion are shown in purple in the revised version; changes from the first revision remain in blue.

[1] V. Pong, S. Gu, M. Dalal, S. Levine. Temporal Difference Models: Model-Free Deep RL for Model-Based Control. ICLR 2018.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.