acceptodds
Under review as a conference paper at ICLR 2027

Covering Behaviors, Not Latents: D-Optimal Curricula for Zero-Shot Reinforcement Learning

Abstract

Zero-shot reinforcement learning aims to learn a reusable policy family before downstream rewards are specified. Recent methods have advanced this goal through predictive representations, which delegate the coverage of behaviors to randomly sampled latent variables or task directions. However, uniform sampling in this index space does not guarantee uniform coverage of the behaviors that matter for transfer. We study this gap through the coverage of policy embeddings, defined by the discounted feature expectations induced by each policy. Motivated by an analytical link between coverage geometry and zero-shot suboptimality, we propose a D-optimal coverage criterion that favors policy embeddings spanning a well-conditioned region of feature space. While leaving each policy’s task-aligned objective unchanged, the corresponding leverage score defines a training curriculum centered on expanding under-covered directions in embedding space. Empirically, we show that D-optimal coverage tracks zero-shot performance and that the resulting curriculum improves learning with both exact and learned policy embeddings, including offline transfer on ExORL, where the gains are most consistent under limited data.

Primary area:
reinforcement learning
Keywords:
zero-shot reinforcement learning, D-optimal coverage

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ihH5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

In zero-shot RL (ZSRL), the authors propose to replace random sampling in task directions with a D-lever sampling curriculum which derives from an optimal coverage criterion which measures the log-volume of the ellipsoid spanned by the policy embeddings. They show in 4 ExoRL environments that D-lever sampling is (somewhat) superior to uniform sampling when used in the Forward-backward (FB) and TD-JEPA algorithms and that it facilitates transfer from an environment to another in the low data regime.

Critical strengths
  • the formalization of the proposed sampling criterion is clean and interesting
Critical weaknesses

** The experimental study suffers from several weaknesses:**

  • the authors compare D-lever to A-lever, E-lever and APS-exact, but A-lever and E-lever are not described anywhere and APS-Exact is hardly described in appendices
  • Figure 1 does not have any variability information, given the small differences in performance it is hard to conclude that these differences are statistically significant
  • Figure 2(a) shows unstable performance under both conditions on Hopper, Figure 2(b) shows more steady progress but no gain from D-lever. The fact that D-lever sampling outperforms uniform sampling in Hopper is questionable, so it is hard to conclude from Fig. 2 that sampling from D-lever brings something. In that respect, Table 1 is more convincing.
  • In 5.4, parag "Coverage expansion under sparse rewards", the main argument of the parag. relies on Table 5, which is in an Appendix, whereas it is unclear what Figure 3 brings.
Questions for authors
  • As D-lever sampling follows a curriculum, it seems more appropriate to leverage it in an online learning than in an offline learning context. On the other hand, FB is offline. Could better explain how D-lever is leveraged in the online and the offline setting? My feeling is that this point deserves being put more forward.
  • Could you explain A-lever and E-lever, please?

Minor remark: this paper looks closely related to [1], which shares the same sampling concern, but in a purely offline context. A comment on that is welcome.

[1] Nazim Bendib, Nicolas Perrin-Gilbert, and Olivier Sigaud. Improving zero-shot offline RL via behavioral task sampling, ICML 2026.

3 comments
AuthorsOpenReview ↗

Authors' Response (1/3)

We are sincerely grateful for the reviewer’s constructive comments and for recognizing the value of our formulation of the sampling criterion. We clarify how the curriculum operates in online and offline learning, explain the comparison methods, and address the experimental results and related work below. We will incorporate these clarifications into the revised manuscript by the end of the discussion period.


D-LEVER in online and offline settings

Q1. As D-lever sampling follows a curriculum, it seems more appropriate to leverage it in an online learning than in an offline learning context. On the other hand, FB is offline. Could better explain how D-lever is leveraged in the online and the offline setting?

D-LEVER is a curriculum over training task directions that allocates more learning effort to latents whose policies contribute to under-covered directions in policy-embedding space. A fixed dataset still allows this allocation to change, since the same transitions can be used to train policies conditioned on different task directions.

In both online and offline learning, we therefore use D-LEVER to sample the latents used for training updates. The sampling distribution is periodically refreshed as the policies and representations evolve, so the curriculum responds to the coverage of the current policy family even when the data are fixed.

In the online setting, we also sample latents from the same curriculum distribution and execute the corresponding policies to collect new transitions. This directs data collection toward under-covered behaviors, providing more data support for the policies that the curriculum prioritizes for improvement. Both settings therefore allocate learning effort according to behavioral coverage, while online interaction additionally enriches the data available for that learning. We will clarify this connection in the main text.


Description of the comparison methods

W1. The authors compare D-lever to A-lever, E-lever and APS-exact, but A-lever and E-lever are not described anywhere and APS-Exact is hardly described in appendices.

Q2. Could you explain A-lever and E-lever, please?

We thank the reviewer for pointing out that these comparisons need a clearer explanation. Full definitions and implementation details are given in Appendix A.3.1, and to make the comparison easier to follow, we summarize the key ideas here. We will also bring this explanation into the main text.

A-LEVER and E-LEVER share the candidate pool, learner, and sampling framework of D-LEVER, and differ only in the coverage criterion. Classical optimal experimental design offers several ways to measure how well a set of embeddings covers the space, each corresponding to a different objective on the regularized second-moment matrix GG of the policy embeddings:

\text{A: }\min\\,\operatorname{tr}(G^{-1}),\qquad \text{E: }\max\\,\lambda_{\min}(G).$$ D-optimality enlarges the overall volume spanned by the embeddings, A-optimality reduces the average uncertainty across directions, and E-optimality targets the single least-covered direction. Each method scores a candidate embedding by how much adding it improves its own objective. Comparing them tests whether the gains come specifically from D-optimality rather than from coverage-based allocation in general. APS-Exact is our tabular adaptation of APS [1] for RiverSwim. Whereas the D-LEVER variants keep each task's reward and only change which tasks are trained, APS changes the reward itself so that policies jointly visit diverse states: $$\text{APS: }\max\\;\underbrace{\mathbb{E}_z\big[\text{return on task } z\big]}_{\text{task}}+\underbrace{\mathcal{H}(\bar d)}_{\text{state diversity}},$$ where $\bar d$ is the state occupancy averaged over a batch of tasks. Original APS estimates the entropy with $k$-nearest neighbors over sampled states; since occupancies in RiverSwim can be computed exactly, APS-Exact uses the exact entropy instead. --- [1] Liu and Abbeel. APS: Active pretraining with successor features. International Conference on Machine Learning (ICML), 2021.
AuthorsOpenReview ↗

Authors' Response (2/3)

Experimental results and interpretation

We thank the reviewer for the comments on our experimental evidence. We report variability across seeds in RiverSwim, provide a task-level breakdown of the Hopper results and discuss the contrast with Ant, and clarify what the Maze trajectories reveal about the curriculum.


Variability in RiverSwim

W2. Figure 1 does not have any variability information, given the small differences in performance it is hard to conclude that these differences are statistically significant.

We agree that variability information should be shown. Figure 1 is based on five seeds, and the variability across seeds is over 10× smaller than the gaps between methods. For example, at the final training step, the mean optimality ratio is 0.9523±0.00010.9523\pm0.0001 for D-LEVER and 0.9473±0.00020.9473\pm0.0002 for Uniform (mean ±\pm standard error across seeds).

Since a single step does not reflect the whole learning curve, we also compare methods by the area under the curve, i.e., each metric averaged over all 21 evaluation steps of training, reported as mean ±\pm standard error across the five seeds.

MetricD-LEVERA-LEVERE-LEVERUniformAPS
Mean0.899 ± 0.0010.881 ± 0.0010.870 ± 0.0020.888 ± 0.0010.882 ± 0.001
CVaR$_{10}$0.796 ± 0.0030.754 ± 0.0030.718 ± 0.0040.766 ± 0.0060.767 ± 0.002
log⁡det⁡G\log\det G25.09 ± 0.1624.50 ± 0.2223.63 ± 0.3523.13 ± 0.1923.20 ± 0.26

D-LEVER achieves the best value on every metric, with gaps that are large relative to the variability across seeds. The gap is smallest for coverage against A-LEVER, which optimizes a closely related criterion. We will add standard-error bands to Figure 1 and report the full statistical tests, including final-step comparisons, in the appendix.


Gains in Hopper task

W3. Figure 2(a) shows unstable performance under both conditions on Hopper, Figure 2(b) shows more steady progress but no gain from D-lever. The fact that D-lever sampling outperforms uniform sampling in Hopper is questionable, so it is hard to conclude from Fig. 2 that sampling from D-lever brings something. In that respect, Table 1 is more convincing.

We agree that the aggregate curves in Figure 2 alone are not conclusive. They average over all preferences, which obscures where D-LEVER helps. To make this clearer, we split the Hopper evaluation preferences into two groups according to which objective receives the larger weight, forward velocity vxv_x or height hh. For each group, the table reports the return of the final policies and the early termination rate, i.e., the fraction of episodes that end before the time limit because the agent falls, as mean ± standard error across seeds.

Hopper, finalUniform (vxv_x)D-LEVER (vxv_x)Uniform (hh)D-LEVER (hh)
Return243.6 ± 16.0238.8 ± 20.2205.6 ± 39.6270.3 ± 8.4
Early termination (%)20.7 ± 10.519.1 ± 6.938.5 ± 14.613.5 ± 6.6

The two methods are similar on forward-velocity-dominant tasks. On height-dominant tasks, however, uniform sampling often terminates early and varies widely across seeds, whereas D-LEVER performs reliably. D-LEVER's gain thus comes from behaviors that uniform sampling under-serves, which is what the coverage criterion targets. On Ant, both objectives concern planar locomotion and uniform sampling already serves them comparably, so no gain is expected, as discussed in Section 5.3. We will add this breakdown to the revision, along with additional seeds.


Visualization under sparse rewards (Figure 3)

W4. In 5.4 parag "Coverage expansion under sparse rewards", the main argument of the parag. relies on Table 5, which is in an Appendix, whereas it is unclear what Figure 3 brings.

We agree that the role of Figure 3 should be clearer and will revise the paragraph accordingly. Figure 3 is meant to show that, although sparse returns may show no progress, the curriculum nonetheless operates as intended.

In Maze, episodes start from the upper left, so goals farther from the start are inevitably less covered. Although the bottom-right goal still yields zero return (Table 5), the leverage scores in Figure 3 nonetheless reflect this ordering, rising from 0.180.18 for the top-left policy to 3.393.39 for the bottom-right one, so D-LEVER directs more training there.

While turning this behavioral coverage into reliable goal-reaching success remains a direction for future work, we are working to further support this observation with quantitative results from training and hope to share them within the discussion period.

AuthorsOpenReview ↗

Authors' Response (3/3)

Related works

Minor remark. this paper looks closely related to [1], which shares the same sampling concern, but in a purely offline context. A comment on that is welcome.

We thank the reviewer for raising this comparison, which helps clarify the position of our work. Both works share the motivation that uniform task sampling is suboptimal, but they pull in nearly opposite directions. Bendib et al. [1] use the offline dataset to identify task directions corresponding to data-supported behaviors and concentrate training on those directions. D-LEVER uses the current policy family as its reference and prioritizes behaviors that remain under-covered. The distinction is therefore between adapting training to the behaviors represented in the data and adapting it to the coverage of the policies being learned. This also explains D-LEVER's applicability to online learning, where the same curriculum can guide data collection to provide additional support for under-covered behaviors.

We also attempted to include [1] as a baseline. As no public implementation was available, we reimplemented the method from the paper. However, we could not reproduce the reported performance despite our efforts, and therefore did not include a comparison based on that implementation. We would be glad to revisit this comparison once a reference implementation becomes available.

These approaches could also be combined. The task distribution learned in [1] could provide a more informative base distribution pZp_Z for D-LEVER by reflecting the behaviors supported by the data. D-LEVER would then adaptively reweight this distribution according to the current policy family's coverage. We will add this discussion to the revised manuscript.


[1] Nazim Bendib, Nicolas Perrin-Gilbert, and Olivier Sigaud. Improving zero-shot offline RL via behavioral task sampling, ICML 2026.

Reviewer_NzZuOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies zero-shot reinforcement learning from the perspective of behavioral coverage. The authors argue that uniformly covering the latent space does not necessarily produce diverse behaviors, since different latent variables may induce highly correlated policies. Based on this observation, the paper proposes D-LEVER, which measures coverage in the learned policy-embedding space using a D-optimality criterion and uses the resulting leverage scores to adaptively allocate training effort toward under-covered behavioral directions. The central insight is intuitive and potentially useful: for zero-shot transfer, the diversity of behaviors induced by the policy family may be more relevant than the diversity of the latent variables used to parameterize that family. The paper evaluates the approach on controlled environments and ExORL and reports improvements over latent-space sampling baselines.

Critical strengths
  1. The paper identifies a meaningful failure mode of latent-space coverage. Different latent codes need not correspond to meaningfully different behaviors, so uniform latent sampling is not necessarily a good proxy for behavioral diversity. This is a useful conceptual distinction for zero-shot RL.

  2. The proposed D-optimality formulation is relatively principled. Rather than relying only on heuristic diversity measures, the method attempts to quantify whether the learned policy family spans sufficiently different behavioral directions. This gives the curriculum a clear geometric interpretation.

  3. The empirical results provide evidence that the proposed strategy can improve zero-shot performance. The inclusion of controlled experiments together with ExORL evaluation makes the empirical story more convincing than a purely toy demonstration.

Critical weaknesses
  1. Limited conceptual novelty. The main intervention combines uniform sampling with leverage-proportional sampling while retaining the underlying RL objectives. Both the determinant criterion and the leverage-based marginal-gain identity are established experimental-design tools. The contribution therefore depends on showing why their application provides a distinctive solution to behavioral coverage in zero-shot RL. However, substantial space is devoted to standard definitions and identities, while the connection to effective neural policy training remains insufficiently established. The exposition consequently gives more attention to inherited machinery than to validating the proposed methodological advance.

  2. Weak coverage justification. The second-moment matrix measures directional spread without retaining sign information, whereas downstream control depends on achieving high returns along specific signed reward directions. Policy families can therefore have identical second moments but substantially different task coverage. The theoretical bound accommodates this discrepancy through an additional approximation gap, yet determinant maximization does not control that gap, and the experiments do not measure it. Consequently, the analysis does not establish that increasing the proposed coverage score improves attainable task performance or reduces actual zero-shot regret.

  3. Unvalidated curriculum rationale. The leverage derivation concerns reallocating probability mass among fixed embeddings, whereas the proposed curriculum reallocates training updates that change embeddings through shared neural parameters. The connection between these settings relies on an idealized embedding update along the latent direction, which is not validated for the actual learner. Moreover, a distinctive existing behavior is not necessarily one that benefits most from additional training. Without measuring learning progress or actual coverage gains from prioritized updates, the justification for using leverage as a training-allocation signal remains incomplete.

  4. Insufficient comparative validation. Comparisons with alternative design criteria and APS are confined to six-state RiverSwim. The continuous-control experiments primarily demonstrate improvements over uniform sampling, with comparable performance on Ant and mixed results under full offline data. These results do not establish that D-optimal geometry is preferable to simpler prioritization based on embedding magnitude, prediction error, or learning progress. Matched comparisons in the learned-embedding setting are needed to distinguish the proposed mechanism from a general benefit of nonuniform training allocation.

  5. Optimistic checkpoint selection. The evaluation selects, separately for each run, the checkpoint with the highest domain-average downstream score. No independent task set for checkpoint selection is specified in the main text. This protocol can produce optimistic estimates and introduces downstream reward information into model selection, even though the policies receive no task-specific gradient updates. Evaluation at predetermined checkpoints, or checkpoint selection using separate validation tasks, is needed to establish whether the reported improvements persist under a practically usable zero-shot deployment protocol.

Questions for authors

See the weaknesses.

5 comments
AuthorsOpenReview ↗

Authors' Response (1/3)

We sincerely thank the reviewer for the careful and thorough review. The detailed comments pointed us to places where the paper did not convey its contribution and evidence as clearly as it should, and working through them has strengthened the paper. We address each concern below and will incorporate these clarifications into the revised manuscript by the end of the discussion period.


Conceptual novelty

W1. …\ldots the determinant criterion and the leverage-based marginal-gain identity are established experimental-design tools. The contribution therefore depends on showing why their application provides a distinctive solution to behavioral coverage in zero-shot RL. …\ldots the connection to effective neural policy training remains insufficiently established$\ldots$

We thank the reviewer for raising this point, which helps us articulate the contribution more clearly. The core contribution of the paper is a principled criterion for behavioral coverage in zero-shot RL, together with a curriculum that achieves it. As the reviewer kindly notes, the formulation quantifies whether the policy family spans sufficiently different behavioral directions rather than relying on heuristic diversity measures, and Theorem 1 connects this coverage to zero-shot suboptimality, a link that has rarely been made explicit in zero-shot RL. This connection is what makes the application of experimental-design tools a distinctive solution to behavioral coverage in zero-shot RL.

The curriculum is accordingly not a standalone sampling intervention but follows from this criterion. To make this link self-contained, we restated the relevant experimental-design tools in this context, and we understand that this may have placed more emphasis on standard material than intended. We thank the reviewer for this observation, and in the revision we will condense this material so that the new elements and their validation stand out more clearly.

We view the connection to neural policy training as an important but secondary aspect of the conceptual contribution, serving to verify that the mechanism holds in practice. In this respect, RiverSwim is more than a toy example, as it isolates the mechanism while all latents share policy parameters. That said, we take the reviewer's point that this evidence can be made more convincing, and we will continue to strengthen it through this response and over the discussion period.


Coverage justification

W2. The second-moment matrix measures directional spread without retaining sign information$\ldots$ The theoretical bound accommodates this discrepancy through an additional approximation gap, yet determinant maximization does not control that gap$\ldots$

The sign information is handled by the policy objective rather than the coverage criterion. Each policy πz\pi_z is trained to maximize z⊤μzz^\top\mu_z, which pushes its embedding toward its own direction, and this is the premise on which any latent-conditioned learner relies. D-LEVER leaves this objective unchanged, unlike reward-augmented approaches such as APS, so the objective is what controls the approximation gap ϵz(w)\epsilon_{z(w)}. D-LEVER is thus not a determinant maximization procedure on its own. As Eq. (18) shows, its reweighted objective equals the average task-aligned utility EpZ[z⊤μz]\mathbb{E}_{p_Z}[z^\top\mu_z] plus a covariance term with leverage (Lemma 4, Appendix C.5), so the determinant enters only through the sampling weights. Consistent with this, fewer than 1% of sampled latents have z⊤μz<0z^\top\mu_z<0 in any of our experiments.

W2. …\ldots Consequently, the analysis does not establish that increasing the proposed coverage score improves attainable task performance or reduces actual zero-shot regret.

The coverage score can be compared across methods only when embeddings lie in a common, fixed feature space, which is why we use RiverSwim as a controlled testbed. Even there, all latents share the same policy parameters, so training one latent also changes the embeddings of the others, and Figure 1 reflects how coverage and transfer evolve under this coupling rather than a trivial setting. For completeness, we report each metric averaged over all 21 evaluation steps of training, as mean ±\pm standard error across five seeds.

MetricD-LEVERA-LEVERE-LEVERUniformAPS
Mean0.899 ± 0.0010.881 ± 0.0010.870 ± 0.0020.888 ± 0.0010.882 ± 0.001
CVaR$_{10}$0.796 ± 0.0030.754 ± 0.0030.718 ± 0.0040.766 ± 0.0060.767 ± 0.002
log⁡det⁡G\log\det G25.09 ± 0.1624.50 ± 0.2223.63 ± 0.3523.13 ± 0.1923.20 ± 0.26

In the Hopper and Ant experiments, the reward features are fixed while the embeddings are learned, so coverage remains comparable across methods. We are rerunning them to track coverage alongside transfer and hope to report the results during the discussion period.

AuthorsOpenReview ↗

Authors' Response (2/3)

Unvalidated curriculum rationale

W3. The leverage derivation concerns reallocating probability mass among fixed embeddings, whereas the proposed curriculum reallocates training updates that change embeddings through shared neural parameters. The connection between these settings relies on an idealized embedding update along the latent direction, which is not validated for the actual learner.

The idealized update μz↦μz+εz\mu_z\mapsto\mu_z+\varepsilon z formalizes the standard premise that training πz\pi_z on its own objective z⊤μzz^\top\mu_z improves that objective, which to first order moves μz\mu_z with a positive component along zz. Proposition 3 relies only on this alignment, with the remainder controlled by the angular mismatch, and does not require the embedding to move exactly along zz.

This holds in our controlled setting even though all latents share the same policy parameters (Appendix A.3.1),

πθ(a∣s,z)∝exp⁡(Ws,a⊤z+bs,a),\pi_\theta(a\mid s,z)\propto\exp\big(W_{s,a}^\top z+b_{s,a}\big),

so that training one latent changes the embeddings of the others. Figure 4 (Appendix A.4) tracks the leverage and optimality ratio of each goal in RiverSwim over training. Leverage first rises on nearer goals and then shifts to farther ones, and each goal's optimality ratio improves shortly after its leverage peaks, after which its leverage declines. Prioritized updates therefore translate into actual gains even under shared parameters.


W3. Moreover, a distinctive existing behavior is not necessarily one that benefits most from additional training. Without measuring learning progress or actual coverage gains from prioritized updates, the justification for using leverage as a training-allocation signal remains incomplete.

We agree that a distinctive behavior is not necessarily the one that benefits most from additional training. How much each latent would benefit, however, is not known in advance. Without such information, assuming that each update moves the embedding by a comparable amount is a natural default, and under this assumption Proposition 3 shows that leverage identifies where an update most increases coverage. Cases in which a prioritized latent cannot be learned at all would equally affect any allocation rule that lacks this information.


Checkpoint selection

W5. The evaluation selects, separately for each run, the checkpoint with the highest domain-average downstream score$\ldots$ This protocol can produce optimistic estimates and introduces downstream reward information into model selection, even though the policies receive no task-specific gradient updates$\ldots$

We agree that this is a fair concern and that we did not explain the rationale for our checkpoint selection in sufficient detail. For the limited-data setting, we follow Jeen et al. [1], the most recent work on zero-shot RL from low-quality offline data, which also provides the VC-FB and MC-FB baselines in Table 1. In this regime, performance fluctuates considerably across checkpoints, so any single predetermined checkpoint gives a noisy estimate. Their protocol therefore selects a single step from the learning curve averaged across seeds and applies it to all runs. In our implementation, however, we selected the checkpoint separately for each run, which deviates from this protocol and can yield optimistic estimates. We have re-evaluated the saved runs with a single step selected from the seed-averaged learning curve, following [1]. Entries report the mean ± standard error across five runs.

DataDomainFBFB + D-LEVERTD-JEPATD-JEPA + D-LEVERMC-FBVC-FB
100KWalker334.8 ± 45.8377.2 ± 34.0387.4 ± 27.9479.8 ± 22.3318.3 ± 56.5346.8 ± 8.7
Quadruped388.4 ± 86.9479.1 ± 74.4590.2 ± 13.8616.7 ± 13.9384.2 ± 122.2246.0 ± 36.3
Maze183.1 ± 75.1254.4 ± 100.4148.4 ± 35.6141.3 ± 38.6311.1 ± 44.8187.6 ± 111.5
Jaco13.0 ± 2.413.0 ± 3.117.5 ± 3.925.5 ± 4.211.3 ± 3.215.3 ± 4.6
FullWalker665.7 ± 14.2667.0 ± 14.0636.4 ± 14.2657.7 ± 4.6––
Quadruped611.7 ± 21.6627.6 ± 7.0670.5 ± 7.5673.7 ± 4.6––
Maze207.6 ± 88.7219.1 ± 85.1163.9 ± 40.4181.8 ± 35.9––
Jaco38.6 ± 11.942.9 ± 7.435.3 ± 8.031.2 ± 8.1––

Under this protocol, D-LEVER improves over the base learner in six of the eight backbone–domain pairs with 100K transitions, with one tie, and in seven of the eight with the full datasets. We will clarify the protocol and its rationale in the main text and update the reported results accordingly.


[1] Jeen et al., Zero-shot reinforcement learning from low quality data. Advances in Neural Information Processing Systems (NeurIPS), 2024.

AuthorsOpenReview ↗

Authors' Response (3/3)

Comparative validation

W4. The continuous-control experiments primarily demonstrate improvements over uniform sampling, with comparable performance on Ant and mixed results under full offline data.

We first address the comparable and mixed results in the learned setting.

On Hopper, we split the evaluation preferences into two groups according to which objective receives the larger weight, forward velocity vxv_x or height hh. For each group, the table reports the return of the final policies and the early termination rate, i.e., the fraction of episodes that end before the time limit because the agent falls, as mean ± standard error across seeds.

Hopper, finalUniform (vxv_x)D-LEVER (vxv_x)Uniform (hh)D-LEVER (hh)
Return243.6 ± 16.0238.8 ± 20.2205.6 ± 39.6270.3 ± 8.4
Early termination (%)20.7 ± 10.519.1 ± 6.938.5 ± 14.613.5 ± 6.6

The two methods are similar on forward-velocity-dominant tasks. On height-dominant tasks, however, uniform sampling often terminates early and varies widely across seeds, whereas D-LEVER performs reliably. D-LEVER's gain thus comes from behaviors that uniform sampling under-serves, which is what the coverage criterion targets. On Ant, both objectives concern planar locomotion and uniform sampling already serves them comparably, so no gain is expected.

With full ExORL data, the base learners appear to already acquire relatively broad behavioral coverage, which leaves less room for reallocating training effort to make a visible difference in standard evaluation. Standard evaluation, however, tests each policy only at the single reward direction inferred from held-out reward samples, z^N=1N∑i=1Nr(si),B(si)\hat z_{N}=\frac{1}{N}\sum_{i=1}^{N}r(s_{i})\\,B(s_{i}), so z^N\hat z_{N} is itself subject to estimation error. Performing well under such error requires coverage of the directions around z^N\hat z_{N}. To test this, we compute z^N\hat z_{N} from NN samples randomly subsampled from the 10,000 used in standard evaluation, using FB and TD-JEPA on Walker with the 10M-transition dataset. This analysis uses the saved checkpoints from our original runs (the best checkpoint of each run, as discussed in W5), and the relative drop compares each checkpoint with itself. For each checkpoint, we average the return over 10 subsamples, and report the mean ±\pm standard error across checkpoints, with the relative drop from N=10,000N=10{,}000 (in %) in parentheses.

NNFBFB + D-LEVERTD-JEPATD-JEPA + D-LEVER
10000698.0698.0706.2706.2672.7672.7679.3679.3
1000649.1±21.2(−7.0±1.1)649.1\pm21.2\\ (-7.0\pm1.1)692.8±4.4(−1.9±0.3)692.8\pm4.4\\ (-1.9\pm0.3)653.9±23.7(−2.8±1.1)653.9\pm23.7\\ (-2.8\pm1.1)673.9±17.6(−0.8±0.8)673.9\pm17.6\\ (-0.8\pm0.8)
500628.9±32.8(−9.9±2.5)628.9\pm32.8\\ (-9.9\pm2.5)673.7±4.2(−4.6±0.9)673.7\pm4.2\\ (-4.6\pm0.9)652.5±16.3(−3.0±0.3)652.5\pm16.3\\ (-3.0\pm0.3)674.5±16.9(−0.7±0.6)674.5\pm16.9\\ (-0.7\pm0.6)
100527.7±16.8(−24.4±1.3)527.7\pm16.8\\ (-24.4\pm1.3)548.0±16.9(−22.4±2.8)548.0\pm16.9\\ (-22.4\pm2.8)579.2±23.8(−13.9±1.9)579.2\pm23.8\\ (-13.9\pm1.9)593.7±31.1(−12.6±3.1)593.7\pm31.1\\ (-12.6\pm3.1)

For both backbones, D-LEVER degrades less as fewer samples are used for reward inference. In both the Hopper and the full-data results, the gains appear where coverage of under-served directions matters, as the geometric criterion predicts.

W4. Insufficient comparative validation. Comparisons with alternative design criteria and APS are confined to six-state RiverSwim. …\ldots These results do not establish that D-optimal geometry is preferable to simpler prioritization based on embedding magnitude, prediction error, or learning progress. Matched comparisons in the learned-embedding setting are needed to distinguish the proposed mechanism from a general benefit of nonuniform training allocation.

We agree that it is important to distinguish the D-optimal geometry from a general benefit of nonuniform training allocation in the learned-embedding setting. To this end, we are implementing simpler prioritization schemes, such as those based on embedding magnitude and prediction error, under the same training setup as D-LEVER, taking care that each is a faithful baseline. We hope to report the results during the discussion period.


We appreciate the reviewer's thorough engagement with our work. We hope these responses clarify the contribution and its validation, and we will share the additional results as they become available during the discussion period. We would be glad to address any further questions.

Reviewer_DDfnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at an interesting issue in zero-shot RL: covering the latent space does not necessarily mean covering diverse behaviors. The authors propose D-LEVER, a curriculum that uses D-optimality and leverage scores to prioritize under-covered behaviors during training. The method can be added to existing RL frameworks without changing their learning objectives. Experiments on RiverSwim, multi-objective control, and ExORL show encouraging results, particularly in low-data settings.

Critical strengths
  1. The motivation behind this work is interesting. The distinction between latent coverage and behavioral coverage is intuitive and provides a useful perspective on zero-shot RL. The proposed method is also simple and easy to integrate into existing frameworks.

  2. The experimental results are encouraging. In particular, the improvements on ExORL with limited data are quite noticeable, and the results across FB and TD-JEPA suggest that the method is not tied to a specific backbone.

Critical weaknesses
  1. I am not fully convinced by the theoretical connection between D-optimal coverage and zero-shot generalization. The bound in Theorem 1 includes an approximation gap that is not explicitly controlled. More importantly, the resulting guarantee appears no tighter than what follows directly from bounded feature expectations. This makes it difficult to see what the theorem tells us beyond the existing assumptions.

  2. I would like to see stronger evidence that D-optimality itself is responsible for the improvements. Since leverage scores depend on both embedding magnitude and direction, a comparison with simple embedding-norm-based sampling would be helpful. Also, the improvements are less consistent when using the full ExORL datasets.

Questions for authors
  1. Could the authors clarify what additional insight Theorem 1 provides beyond the direct bound from bounded feature expectations? Is there a way to control the approximation gap so that the guarantee becomes more informative?

  2. Have the authors considered comparing D-LEVER with embedding-norm-based sampling on ExORL? I think this would help clarify whether the gains mainly come from better directional coverage or simply from prioritizing larger embeddings.

I am open to reconsidering my rating if the authors can address these concerns in the rebuttal.

5 comments
AuthorsOpenReview ↗

Authors' Response (1/2)

We thank the reviewer for the thoughtful comments and for recognizing the value of our approach and its empirical results. We address the comments in three parts, covering the interpretation of Theorem 1 and its approximation gap, the roles of embedding magnitude and direction in the leverage score, and the full-data ExORL results. We will carefully incorporate the clarifications and additional results presented in this response into the revised manuscript by the end of the discussion period.


Theoretical concerns (Theorem 1)

We appreciate the reviewer’s close reading of Theorem 1. We clarify its interpretation relative to the bounded-feature bound, then explain how the approximation gap motivates our algorithmic design.

W1. More importantly, the resulting guarantee appears no tighter than what follows directly from bounded feature expectations.

Q1. Could the authors clarify what additional insight Theorem 1 provides beyond the direct bound from bounded feature expectations?

Under Assumption 1, every policy embedding satisfies ∣μπ∣2≤L=Φmax⁡/(1−γ)\\|\mu_{\pi}\\|_{2} \leq L = \Phi_{\max}/(1-\gamma), where Φmax⁡\Phi_{\max} bounds the feature norm and γ\gamma is the discount factor. With ∣w∣2=1\\|w\\|_{2} = 1, this immediately gives

J(πw∗;w)−J(πz(w);w)≤L−w⊤μz(w).J(\pi^{*}_{w};w) - J(\pi_{z(w)};w) \leq L - w^{\top}\mu_{z(w)}.

Theorem 1 states that, with probability at least 1−δ1-\delta over test directions,

J(πw∗;w)−J(πz(w);w)≤(L−(δdet⁡GpZκ)1/d)++ϵz(w),J(\pi^{*}_{w};w) - J(\pi_{z(w)};w) \leq \left(L - \left(\frac{\delta\sqrt{\det G_{p_{Z}}}}{\kappa}\right)^{1/d}\right)_{+} + \epsilon_{z(w)},

where GpZG_{p_{Z}} is the regularized second-moment matrix of policy embeddings under the base latent distribution, dd is the feature dimension, δ\delta is the failure probability, and κ\kappa limits the concentration of the test-task distribution.

Theorem 1 is not intended as a tighter absolute guarantee for a single policy family. Its purpose is comparative, in that it ranks policy families by coverage. The first term is decreasing in log⁡det⁡GpZ\log\det G_{p_{Z}}, so among policy families with comparable approximation gap, the one with larger D-optimal coverage receives a smaller high-probability bound on zero-shot suboptimality.

In particular, the same argument gives a coverage-dependent lower bound on the selected policy's return,

J(πz(w);w)≥(δdet⁡GpZκ)1/d−ϵz(w),J(\pi_{z(w)};w) \geq \left(\frac{\delta\sqrt{\det G_{p_{Z}}}}{\kappa}\right)^{1/d} - \epsilon_{z(w)},

which separates two sources of suboptimality, namely geometric under-coverage of the family captured by det⁡GpZ\det G_{p_{Z}}, and the approximation gap of the selected policy ϵz(w)\epsilon_{z(w)}. The bound from bounded features alone does not distinguish these two.

W1. The bound in Theorem 1 includes an approximation gap that is not explicitly controlled.

Q1. Is there a way to control the approximation gap so that the guarantee becomes more informative?

The comparative reading above holds when ϵz(w)=(w⊤GpZw−w⊤μz(w))+\epsilon_{z(w)} = \left(\sqrt{w^{\top} G_{p_{Z}} w} - w^{\top}\mu_{z(w)}\right)_{+} is small, and this is exactly what determines our algorithmic design. Expanding coverage is useful only if the selected policy's task return keeps pace with the coverage proxy. We therefore keep each policy's task-aligned objective J(πz;z)=z⊤μzJ(\pi_{z};z) = z^{\top}\mu_{z} (Eq. 15) unchanged and use leverage scores only to reallocate training frequency. Adding a coverage bonus to the reward, as in APS-style methods, could instead steer πz\pi_{z} away from the behavior prescribed by zz, enlarging the gap while increasing the proxy.

Fix a task direction ww with ∣w∣2=1\\|w\\|_{2}=1 and a finite policy family. The coverage proxy is w⊤GpZw=∑zpZ(z),(w⊤μz)2+λw^{\top} G_{p_{Z}} w = \sum_{z} p_{Z}(z)\\,(w^{\top}\mu_{z})^{2} + \lambda. Let v=w⊤μz(w)v = w^{\top}\mu_{z(w)} and pp be its mass under the base distribution. Holding the other contributions and λ\lambda fixed as CC, the gap becomes

ϵz(w)=[C+pv2−v]+.\epsilon_{z(w)}=\left[\sqrt{C+pv^{2}}-v\right]_{+}.

Wherever the gap is positive,

∂ϵz(w)∂v=pvC+pv2−1≤p−1<0(p<1).\frac{\partial\epsilon_{z(w)}}{\partial v} = \frac{pv}{\sqrt{C+pv^{2}}}-1 \leq \sqrt{p}-1 < 0 \quad (p<1).

Increasing the selected policy's task return therefore reduces a positive gap, even accounting for its contribution to the coverage proxy. When latent and task directions match, the task-aligned objective directly drives this increase, motivating D-LEVER’s use of leverage scores to change training frequency while preserving the per-policy objective.

This analysis fixes the remaining embeddings, which move jointly under shared-parameter training. A training-time guarantee would require analyzing these joint dynamics, an open problem also faced by FB and TD-JEPA. While such a guarantee is beyond our claims, we hope this analysis provides a useful rationale for improving coverage while retaining the task-aligned objective to control the approximation gap.

AuthorsOpenReview ↗

Authors' Response (2/2)

Magnitude vs. direction in leverage

W2. I would like to see stronger evidence that D-optimality itself is responsible for the improvements. Since leverage scores depend on both embedding magnitude and direction, a comparison with simple embedding-norm-based sampling would be helpful.

Q2. Have the authors considered comparing D-LEVER with embedding-norm-based sampling on ExORL? I think this would help clarify whether the gains mainly come from better directional coverage or simply from prioritizing larger embeddings.

We thank the reviewer for this question, which helps clarify what drives D-LEVER’s sampling behavior. Embedding-norm-based sampling is in fact a limiting case of D-LEVER itself. As shown in Appendix E, as the ridge λ→∞\lambda\to\infty, the leverage-proportional component converges to sampling in proportion to ∥μz∥22\lVert\mu_{z}\rVert_{2}^{2}. While a training comparison with embedding-norm-based sampling is in progress, we report a post-hoc analysis of how magnitude and direction each shape D-LEVER's sampling distribution.

Using saved ExORL Walker checkpoints, we decomposed D-LEVER's leverage score into magnitude and directional factors as follows:

ℓ(z)=ℓnorm(z),ℓdirection(z),ℓnorm(z)=∥μz∥22,ℓdirection(z)=μz⊤G−1μz/∥μz∥22.\ell(z)=\ell_{\mathrm{norm}}(z)\\,\ell_{\mathrm{direction}}(z),\qquad\ell_{\mathrm{norm}}(z)=\lVert\mu_{z}\rVert_{2}^{2},\qquad\ell_{\mathrm{direction}}(z)=\mu_{z}^{\top}G^{-1}\mu_{z} / \lVert\mu_{z}\rVert_{2}^2.

For each checkpoint, we averaged the diagnostic over five candidate pools. We then report the mean and standard error across five independently trained checkpoints.

BackboneCorr$(\ell,\ell_{\mathrm{norm}})$Corr$(\ell,\ell_{\mathrm{direction}})$TV after removing ℓnorm\ell_{\mathrm{norm}}TV after removing ℓdirection\ell_{\mathrm{direction}}
FB0.056±0.0700.056\pm0.0700.454±0.0540.454\pm0.05424.9±2.524.9\pm2.5%38.5±3.138.5\pm3.1%
TD-JEPA0.167±0.2510.167\pm0.2510.226±0.2620.226\pm0.26227.7±9.527.7\pm9.5%30.9±7.930.9\pm7.9%

Across both backbones, removing either factor reallocates roughly a quarter or more of the sampling mass, indicating that both factors shape D-LEVER's sampling behavior. These diagnostics show that D-LEVER does more than prioritize larger embeddings. We are also training with embedding-norm-based sampling under the same setup and hope to report the results during the discussion period if time permits.


Full-data ExORL results

W2. Also, the improvements are less consistent when using the full ExORL datasets.

We thank the reviewer for this observation and present below an additional analysis showing a benefit of D-LEVER that standard full-data evaluation does not capture. With sufficient data, the base learners appear to already acquire relatively broad behavioral coverage, which leaves less room for reallocating training effort to make a visible difference in standard evaluation.

Standard evaluation, however, tests each policy only at the single reward direction inferred from held-out reward samples, z^N=1N∑i=1Nr(si),B(si)\hat z_{N}=\frac{1}{N}\sum_{i=1}^{N}r(s_{i})\\,B(s_{i}), so z^N\hat z_{N} is itself subject to estimation error. Performing well under such error requires coverage of the directions around z^N\hat z_{N}. To test this, we compute z^N\hat z_{N} from NN samples randomly subsampled from the 10,000 used in standard evaluation. We use FB and TD-JEPA on Walker with the 10M-transition dataset. For each checkpoint, we average the return over 10 subsamples, and report the mean ±\pm standard error across checkpoints, with the relative drop from N=10,000N=10{,}000 (in %) in parentheses.

NNFBFB + D-LEVERTD-JEPATD-JEPA + D-LEVER
10000698.0698.0706.2706.2672.7672.7679.3679.3
1000649.1±21.2 (−7.0±1.1)649.1\pm21.2\ (-7.0\pm1.1)692.8±4.4 (−1.9±0.3)692.8\pm4.4\ (-1.9\pm0.3)653.9±23.7 (−2.8±1.1)653.9\pm23.7\ (-2.8\pm1.1)673.9±17.6 (−0.8±0.8)673.9\pm17.6\ (-0.8\pm0.8)
500628.9±32.8 (−9.9±2.5)628.9\pm32.8\ (-9.9\pm2.5)673.7±4.2 (−4.6±0.9)673.7\pm4.2\ (-4.6\pm0.9)652.5±16.3 (−3.0±0.3)652.5\pm16.3\ (-3.0\pm0.3)674.5±16.9 (−0.7±0.6)674.5\pm16.9\ (-0.7\pm0.6)
100527.7±16.8 (−24.4±1.3)527.7\pm16.8\ (-24.4\pm1.3)548.0±16.9 (−22.4±2.8)548.0\pm16.9\ (-22.4\pm2.8)579.2±23.8 (−13.9±1.9)579.2\pm23.8\ (-13.9\pm1.9)593.7±31.1 (−12.6±3.1)593.7\pm31.1\ (-12.6\pm3.1)

For both backbones, D-LEVER performs comparably to the base method with all 10,000 reward samples, but degrades less as fewer samples are used for reward inference. When the inferred direction is less accurate, broader coverage of nearby directions makes it more likely that the selected policy still retains useful behavior, which matches the behavioral-coverage motivation. Thus, even when standard evaluation shows similar performance, D-LEVER provides an additional benefit in the full-data setting in the form of greater robustness to errors in the inferred reward direction.


We hope these responses address the reviewer's concerns, and we welcome any further questions or feedback.

AuthorsOpenReview ↗

Summary of Revisions Addressing Reviewer DDfn’s Comments

We thank the reviewer for the constructive feedback. We have uploaded the revised manuscript, and below we describe how it addresses each comment, followed by other major changes made for the other reviewers.


Theorem 1 and the bounded-feature bound (W1, Q1)

We have restated Theorem 1 as a high-probability lower bound on the return of the selected policy (Sec. 3, p. 4),

J(πz(w);w)≥(δdet⁡GpZκ)1/d−ϵz(w).J(\pi_{z(w)};w) \geq \left(\frac{\delta\sqrt{\det G_{p_{Z}}}}{\kappa}\right)^{1/d} - \epsilon_{z(w)}.

In the original version, the coverage term appeared next to L=Φmax⁡/(1−γ)L = \Phi_{\max}/(1-\gamma), which made the result look like a refinement of the bounded-feature bound. The revised statement no longer involves LL or the optimal policy, and states directly what coverage guarantees about zero-shot return. Since it holds with high probability over tasks, we now evaluate transfer by VaR, the return level attained by a given fraction of tasks (Sec. 5, p. 6).

The gap in Definition 2 is now signed (Sec. 3, p. 4), and Lemma 1 gives the identity

J(πz(w);w)=w⊤GpZw−ϵz(w),J(\pi_{z(w)};w) = \sqrt{w^{\top} G_{p_{Z}} w} - \epsilon_{z(w)},

so the return splits into a coverage term, which Theorem 1 bounds through det⁡GpZ\det G_{p_{Z}}, and the gap of the selected policy (App. E.1, p. 28).


Approximation gap (W1, Q1)

Section 3 now notes that the bound improves with larger coverage and a smaller gap (Sec. 3, p. 4). This is why D-LEVER keeps each policy's task-aligned objective unchanged and uses leverage scores only to decide how often each latent is trained (Sec. 4.2, p. 5).

A training-time guarantee on the gap would require analyzing the joint dynamics of embeddings under shared parameters, which remains beyond our claims. We instead examine this in RiverSwim, where all policies share parameters while embeddings are computed exactly, so the effect of shared-parameter training on the gap can be observed without estimation noise. D-LEVER achieves the highest coverage and the lowest mean gap through most of training (Fig. 1, p. 7), along with the highest VaR at every reported fraction (Table 1, p. 7).


Embedding magnitude and direction (W2, Q2)

We added two baselines (Sec. 5.2, p. 9), which replace the post-hoc analysis in our earlier response: NORM, which samples latents in proportion to embedding magnitude, and TD, which samples in proportion to temporal-difference error. The former tests whether prioritizing larger embeddings is enough, and the latter whether the gains come from nonuniform allocation in general.

On ExORL, D-LEVER outperforms both baselines in most backbone–domain pairs, while NORM and TD seldom improve on the base learner (Table 3, p. 9). In multi-objective control, NORM is competitive on Hopper but not on Ant, and TD shows the opposite pattern (Table 2, p. 8). D-LEVER improves over uniform sampling in both domains and achieves the highest returns along the requested directions. Neither magnitude nor prediction error alone explains the gains.


Full-data ExORL results (W2)

We agree that the gains with the full datasets are limited.

The main offline experiment focuses on the limited-data setting, where training allocation is expected to matter most (Sec. 5.3, p. 9). Full-data results under the same protocol are in the appendix (Table 8, p. 27). They differ from those in our earlier response because the revised checkpoint protocol is applied throughout and TD-JEPA uses a learning rate selected by a sweep.

We also discuss why the gains shrink with more data (App. A, p. 13). Since the prediction targets of these learners depend on the policy, limited data may support some behaviors better than others, leaving more room for reallocating updates. With abundant data, the question shifts to whether the sampled task directions lead policies to exploit the diverse behaviors in the dataset, which [2] addresses and is complementary to D-LEVER.

We did not include the reward-subsampling analysis from our earlier response, as we found it inconclusive.


Other major changes

  • Checkpoint selection follows [1], with Top-5 results added (Table 3, p. 9). (Reviewer NzZu)
  • Behavioral coverage in continuous control is visualized (Fig. 2, p. 8). (Reviewers NzZu & ihH5)
  • A-LEVER, E-LEVER, and APS-Exact are described in the main text (Sec. 5.1, p. 7). (Reviewer ihH5)
  • Standard errors are added to the RiverSwim results (Fig. 1 and Table 1, p. 7). (Reviewer ihH5)
  • D-LEVER in online and offline settings is explained (Sec. 4.3, p. 6). (Reviewer ihH5)
  • The role of Figure 3 is clarified (Sec. 5.3, p. 10). (Reviewer ihH5)

We thank the reviewer again for the helpful comments, and we would be glad to answer any further questions.


[1] Jeen et al., Zero-shot reinforcement learning from low quality data. NeurIPS 2024.

[2] Bendib et al., Improving zero-shot offline RL via behavioral task sampling. ICML 2026.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Summary of Revisions Addressing Reviewer ihH5's Comments

We sincerely thank the reviewer for taking the time to read our responses and for the encouraging follow-up, which we greatly appreciated. The revised manuscript is now uploaded, and below we describe how it addresses each comment, followed by other major changes made for the other reviewers.


Comparison methods (W1, Q2)

A-LEVER, E-LEVER, and APS-Exact are now described in the main text (Section 5.1, page 7). The three curricula share the candidate pool, learner, and resampling framework, and differ only in the design criterion applied to the regularized second-moment matrix GG of the policy embeddings: D-optimality maximizes log⁡det⁡G\log\det G, A-optimality minimizes tr(G−1)\mathrm{tr}(G^{-1}), and E-optimality maximizes λmin⁡(G)\lambda_{\min}(G). APS-Exact adds an exactly computed state-occupancy entropy bonus to the task reward. See (Appendix C.1, page 19).


RiverSwim results (W2)

Figure 1 now shows standard-error bands (Figure 1, page 7). Table 1 reports training-averaged and final-policy results with standard errors (Table 1, page 7). D-LEVER attains the highest value on nearly every metric. Since the differences are large relative to the standard errors, we report these directly rather than separate statistical tests.

The evaluation now also follows Theorem 1. Since the theorem bounds return with high probability over tasks, we evaluate transfer by VaR, the return level attained by a given fraction of tasks, with CVaR as a secondary metric (Section 5, page 6). We also report the approximation gap that appears in the theorem, which shows how informative its coverage-based bound is in practice (Figure 1, page 7). D-LEVER attains the highest VaR at every reported fraction and the lowest gap through most of training.


Online and offline settings (Q1)

We added a paragraph on how D-LEVER operates in the two settings (Section 4.3, page 6). In both, D-LEVER selects the latents used for learner updates, which can change even with a fixed dataset since the same transitions can train policies for different task directions. For the online setting, the revision differs from our earlier response. With the added baselines, collecting data with each method's own sampling rule made training unstable and the comparison hard to interpret. All methods therefore collect data with latents from the base distribution, and the sampling rules differ only in learner updates (Appendix C.2, page 21). The curriculum still shapes the data indirectly, since policies that cover a broader range of behaviors collect more diverse experience.


Hopper and Ant results (W3)

We agree that the aggregate learning curves were not conclusive. With the setup above, the experiment is rerun and reorganized. The curves are moved to the appendix (Figure 6, page 24), and the main text now reports the mean, VaR, and CVaR of the final policies with standard errors (Table 2, page 8). Figure 2 replaces the two-group breakdown in our earlier response, showing the behaviors the final policies reach across all evaluation tasks (Figure 2, page 8). On Hopper, D-LEVER reaches a broader range of behaviors with much higher and more stable VaR. On Ant, the methods reach similar ranges, and D-LEVER still improves over uniform sampling.


Role of Figure 3 (W4)

The paragraph is rewritten so that its argument rests on Figure 3 itself (Section 5.3, page 10). The figure shows that the curriculum operates as intended even when sparse returns show no progress: the bottom-right policy receives zero return, yet its high leverage directs more training toward it. The additional quantitative analysis we had hoped to share was not conclusive, so we do not include it and state that turning this coverage into reliable goal-reaching remains future work.


Related work (Minor remark)

We added a discussion of [1] (Appendix A, page 13). Their Behavioral Task Distribution constructs the task distribution from dataset behaviors, which informs the choice of pZp_{Z}, while D-LEVER reallocates updates according to the coverage of the current policy family. The two are thus complementary.


Other major changes

  • Theorem 1 is restated as a lower bound on zero-shot return (Section 3, page 4). (Reviewers DDfn & NzZu)
  • Sampling by embedding magnitude and by TD error are added as baselines (Table 2, page 8, and Table 3, page 9). (Reviewers DDfn & NzZu)
  • ExORL checkpoint selection follows [2], with the top-five checkpoint average added (Table 3, page 9). Numbers differ from the original version for this reason and a TD-JEPA learning-rate sweep. (Reviewer NzZu)

We thank the reviewer again for the helpful comments, and we would be glad to address any remaining concerns during the discussion period.


[1] Bendib et al., Improving zero-shot offline RL via behavioral task sampling. ICML 2026.

[2] Jeen et al., Zero-shot reinforcement learning from low quality data. NeurIPS 2024.

AuthorsOpenReview ↗

Summary of Our Responses and Revisions

We thank all reviewers for their feedback, which showed us where the original presentation left our main points unclear. This comment gathers our individual replies in one place, organized by the paper's contributions, listing the concerns raised and how the revised manuscript addresses each. As the discussion period is drawing to a close, we would be grateful to hear whether these responses address your concerns.


Theoretical guarantee

C1. What Theorem 1 adds to the bounded-feature bound (DDfn W1, Q1)

Stated as a lower bound on return. The original statement was J(πw∗;w)−J(πz(w);w)≤(L−c)++ϵz(w)J(\pi^{*}_{w};w) - J(\pi_{z(w)};w) \leq (L - c)_{+} + \epsilon_{z(w)}, with cc the coverage term and LL the bounded-feature constant. In this form the theorem read as claiming a tight bound for a particular policy family, which bounded features already give. The revised statement gives the return level c−ϵz(w)c - \epsilon_{z(w)} that a family guarantees across tasks (Sec. 3). Thus, it is to be read in a comparative manner, across policy families with different coverage. The original form adds the bounded-feature limit on the optimal return, so the two differ only in presentation.

C2. Lost sign information left to ϵz(w)\epsilon_{z(w)} (NzZu W2)

ϵz(w)\epsilon_{z(w)} is now signed. As Reviewer NzZu notes, the second moment drops the sign of returns, and the theorem accounts for this through ϵz(w)\epsilon_{z(w)}. The revision no longer clips ϵz(w)\epsilon_{z(w)} at zero (Def. 2). In either case, D-LEVER takes ϵz(w)\epsilon_{z(w)} into account by keeping each policy trained on its own task return, instead of maximizing the determinant alone.

C3. Whether ϵz(w)\epsilon_{z(w)} is controlled (DDfn W1, Q1 and NzZu W2, W3)

By algorithmic design; now measured in training. ϵz(w)\epsilon_{z(w)} compares the policy selected for task ww against the family on that same task. Training each πz\pi_{z} on the reward z⊤ϕ(s,a)z^{\top}\phi(s,a) makes it stand out on its own task, and thus D-LEVER leaves this objective unchanged and only reallocates training, as in Sec. 4.2. While a theoretical guarantee during training is beyond the scope of this work, we examine ϵz(w)\epsilon_{z(w)} empirically in RiverSwim, where policies share parameters, and find that it decreases as D-LEVER expands coverage (Fig. 1).


Curriculum rationale

C4. Emphasis on standard experimental-design material (NzZu W1)

The standard results were included to keep the paper self-contained. They are moved to the appendix, and the revision puts more effort into showing empirically how the curriculum behaves (Sec. 5).

C5. Fixed-embedding derivation, and lack of coverage evidence (NzZu W3)

Update along zz via the reward z⊤ϕz^{\top}\phi; coverage gains empirically shown. The exact update μz↦μz+εz\mu_{z} \mapsto \mu_{z} + \varepsilon z in the derivation is illustrative, and as Prop. 3 (App. E.3) shows, it is enough that zz and μz\mu_{z} form an acute angle, which is what training πz\pi_{z} on the reward z⊤ϕ(s,a)z^{\top}\phi(s,a) works toward. In RiverSwim, where policies share parameters, the training-averaged log⁡det⁡G\log\det G is highest for D-LEVER (Table 1), and with neural policies in Hopper and Ant the gain appears in the behaviors reached and in the returns (Fig. 2 and Table 2).

C6. Use in online and offline settings (ihH5 Q1)

In either case, each learner update pairs a batch of transitions with task latents, and there D-LEVER decides which latents are used for training more often. It therefore acts in the same way online and offline (Sec. 4.3, p. 6), and online it also shapes the data, since the replay buffer is filled by the policy family it trains.


Empirical evidence

C7. Theorem 1 in practice (NzZu W2, W3 and ihH5 W3)

  • We now report VaR as the main metric, namely the return level that a given fraction of tasks attains, as the metric closest to what Theorem 1 states (Sec. 5).
  • In RiverSwim, performance rises with coverage while ϵz(w)\epsilon_{z(w)} falls (Fig. 1).
  • On Hopper and Ant, D-LEVER reaches a broader set of behaviors (Fig. 2), and this comes with strong performance (Table 2).

C8. Alternative sampling baselines (DDfn W2, Q2 and NzZu W4)

  • We added baselines that sample by embedding magnitude and by prediction error. Each helps in some settings, but neither does so across them (Tables 2 and 3).

C9. Reliability of the results (NzZu W5 and DDfn W2)

  • Results are now reported as exact values with standard errors, so that differences and their variability can be checked directly (Tables 1 and 2).
  • Checkpoints now follow the protocol of [1] and are reported with the top-five average per run (Sec. 5.3).
  • The limited gains with the full datasets are acknowledged and discussed (App. A).

[1] Jeen et al., Zero-shot reinforcement learning from low quality data. NeurIPS 2024.

Could the authors quickly highlight the changes (e.g. in blue) they made in the revised version ? Unfortunately I won't have time to read the full paper again before the discussion period ends. Sorry for the late reaction...

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.