- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces Offline Quality‑Diversity, a new reinforcement learning problem setting. Unlike traditional online Quality Diversity, which needs real‑environment interaction to evaluate behaviors, Offline QD aims to build a full QD policy archive only from static offline datasets, with no environment access during training. Given predefined behavioral descriptors, it seeks high performance policies that match each target behavior. Since existing offline methods require extra online rollouts to map onto target descriptors, they cannot solve Offline QD. To preliminarily verify the tractability of the problem, the authors adapt three established offline RL paradigms to tackle it. Experiments show these methods work across locomotion, navigation and manipulation tasks. All three methods have clear inherent limitations from out‑of‑distribution risks, model drift or dataset coverage. However, the analysis of the challenges unique to this setting remains somewhat limited.
Critical strengths
-
Offline-QDAC leverages successor features (SF) as a proxy for behavioral compliance and is grounded in theoretical analysis. Despite a gap between theory and practical implementation arising from replacing the policy’s state distribution with the dataset distribution, the theoretical framework still helps us understand the method’s regime of validity.
-
To address the issue that prior offline QD methods cannot be deployed in a truly fully offline fashion, the paper defines a more stringent offline QD problem formulation. This represents the paper’s primary contribution, and the work presents several basic solutions tailored to this new problem setting.
-
The experimental validation is relatively comprehensive. It verifies that the proposed problem formulation is indeed tractable under several common experimental setups, and outperforms online algorithms in certain cases.
Critical weaknesses
-
If the SF proxy compliance signal steps into out-of-distribution (OOD) regions, it becomes unreliable. The authors did not try to offer any effective safeguards.
-
The three proposed solutions mainly adapt three existing paradigms to the new problem. The paper lacks sufficient analysis of the novel challenges introduced by this setting and does not provide targeted innovative ideas to address them.
Questions for authors
In the standard offline RL setting, OOD extrapolation error primarily concerns scalar rewards. In the offline QD setting, however, the BD becomes a vector, and the corresponding SF function outputs a vector as well. This raises the question: is controlling OOD extrapolation error in this scenario more challenging than controlling extrapolation error for the Q-function in conventional offline RL?
4 comments
Response to Reviewer R5Dp (2/2): Offline QD challenges, SF extrapolation
W2 (novelty and Offline QD challenges). We agree that our three methods adapt existing paradigms. However, we believe the paper and this rebuttal identify several challenges that do not arise in standard offline RL, together with how we address each. In the revised version, we gather them in Section 6 ("Challenges and Limitations"):
- Behavior cannot be verified offline. Checking whether a policy achieves its target descriptor requires a rollout for every target in the descriptor space. Our successor-feature surrogate provides an offline signal, and Corollary 1 makes its dependence on dataset coverage explicit.
- Data must cover the descriptor space. In offline RL we need data near high-return behavior, but in Offline QD we also need data across all target behaviors. Dataset quality thus has two competing axes: App. E.4 shows that QD score peaks at an intermediate mix of fitness-oriented and diversity-oriented data.
- Descriptors are trajectory-level. Offline goal-conditioned RL reuses data by relabeling any future state as a goal. A QD descriptor characterizes a whole episode, so this relabeling does not directly apply. Successor features address this for additive descriptors by predicting the future discounted average of features.
- Fitness and compliance conflict. A weighted sum lets fitness override compliance; our gate enforces compliance first. A weighted goal-plus-fitness baseline (GCIQL + fitness) reaches a lower QD score than Offline QDAC (527 vs. 622 on Antmaze, 211 vs. 261 on Cube).
- Imagined search must cover full episodes. Without replanning from real states, world-model errors compound over the horizon, a structural limitation rather than a capacity issue (App. E.9).
We see this characterization as a starting point for methods designed specifically for Offline QD.
Q1 (extrapolation error for successor features). In our view, the difficulty does not come from the SF being a vector, but from what the SF must represent. The same difficulty would arise with a one-dimensional feature.
Why in-sample learning works for but not for the SF.
- can be learned from dataset actions only. IQL estimates the value of the best behavior in the data. This only needs actions present in the data, so no unseen action is queried.
- The SF cannot. The SF must describe the current policy , because it measures whether this policy reaches its target . Learned from dataset actions only, it describes the behaviors in the data instead of . To evaluate , the SF targets must use actions of , and these actions can be out of distribution.
- Evidence. -IQL (W1) learns the SF in-sample, as IQL does for . It avoids unseen actions entirely, yet performs worst in Table R2 (514 vs. 622 on Antmaze), since its SF do not describe the current policy (App. E.6).
Why SF errors are easier to control in one respect. The SF update has no max over actions, and the true SF is bounded by . SF errors are therefore not amplified by a max operator, as Q-learning errors are. The remaining errors are limited by the safeguards of W1, in particular the BC term on Antmaze and Cube.
[1] S. Park, K. Frans, B. Eysenbach, S. Levine. OGBench: Benchmarking Offline Goal-Conditioned RL. ICLR 2025.
[2] S. Fujimoto, S. S. Gu. A Minimalist Approach to Offline Reinforcement Learning. NeurIPS 2021.
[3] S. Park, K. Frans, S. Levine, A. Kumar. Is Value Learning Really the Main Bottleneck in Offline RL? NeurIPS 2024.
[4] X. B. Peng, A. Kumar, G. Zhang, S. Levine. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. arXiv:1910.00177, 2019.
[5] I. Kostrikov, A. Nair, S. Levine. Offline Reinforcement Learning with Implicit Q-Learning. ICLR 2022.
Follow-up on W2: a mechanism specific to Offline QD
Regarding W2 (lack of targeted ideas for the challenges of the setting), our new global comment describes a mechanism designed for one such challenge: estimating, from offline data only, whether a policy will reach a target descriptor that is not additive, such as a final position or a range of motion. The new critic fixes the failure of Offline QDAC on the range descriptor (487 → 1896), and clearly improves it on Antmaze Explore (+60%) and Cube Play (+36%). We would be grateful to know whether this addresses the reviewer's concern.
Official Review of Submission11423 by Reviewer R5Dp
Thank you to the authors for the considerable effort they have devoted to responding to the review questions. I also partly agree with several other reviewers regarding the limited methodological novelty. If the formulation of a new problem such as offline QD is to be treated as an important contribution, then the unique new challenges that such a new problem faces need to be articulated much more clearly. Although the authors do enumerate some relevant challenges in their response, challenges such as "Behavior cannot be verified offline", or "Data must cover the descriptor space", are not problems that any algorithm can solve — and the offline data available in practice is often difficult to control deliberately. I therefore believe that the new problems and new challenges in offline QD that are genuinely solvable by algorithms are indeed still not as clearly delineated as the core challenge identified for offline RL at the time, namely OOD extrapolation error. Overall, most reviewers agree that this problem is worth exploring, and the authors have indeed been the first to do a substantial amount of foundational work in this direction. I maintain my rating unchanged.
Response to Reviewer R5Dp (1/2): OOD safeguards
We thank the reviewer for the constructive review, and for recognizing the Offline QD problem as our primary contribution, the theoretical grounding of the SF proxy, and the comprehensive experimental validation. In response, we (i) detail the safeguards against OOD actions and evaluate two stronger ones, AWR and in-sample SF learning (W1, Tables R1-R2), (ii) gather the challenges specific to Offline QD in Section 6 (W2), and (iii) discuss what makes extrapolation error harder to control for successor features (Q1). All numbers refer to the revised version; changes are shown in blue.
W1 (OOD safeguards). We agree. IQL's in-sample training protects the critic targets, but the actor and the SF targets query actions generated by the policy. Four mechanisms limit extrapolation in practice:
Effect of the BC term. The constraint is essential on the OGBench tasks, but it reduces performance on locomotion (Table R1). We therefore apply it only to Antmaze and Cube.
Table R1: Effect of the BC term on Offline QDAC, averaged over datasets per environment (mean ± std, 5 seeds). Table 16 in the revised paper.
Alternatives to our design. We also compare three variants, each replacing one component of Offline QDAC (App. E.6, Table 15 of the revised version):
LE=E(s,a)∼D,z[τ−1[u<0]ψ(s,a,z)−E(s,z)22],u=(1−γ)E(s,z)−z−(1−γ)ψ(s,a,z)−z,
and bootstrap the successor features through it, ψ(s,a,z)←ϕ(s)+γE(s′,z), so that no policy action is queried. As τ→1, E(s,z) approaches the successor features of the most compliant dataset action, ψ(s,a⋆,z) with $a^\star = \arg\min_{a \in \mathrm{supp} \mathcal{D}(\cdot|s)} \Vert (1-\gamma)\psi(s,a,z) - z\Vert $.
Table R2: Offline QDAC with alternative policy extraction and SF learning, averaged over datasets per environment (mean ± std, 5 seeds). Table 15 in the revised paper.
Our design performs best in all four environments. Removing the conservative critic (TD3) lowers performance, except on Walker, where all variants overlap within one standard deviation. AWR is consistently weaker, in line with prior findings that weighted regression extracts less from a learned value than gradient-based extraction [3]. ψ-IQL performs worst in all environments; we explain why in our answer to Q1.
Finally, the disagreement between the two heads of our SF network could provide an uncertainty estimate to penalize the actor. We mention this direction in Section 6 of the revised version.