- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In zero-shot RL (ZSRL), the authors propose to replace random sampling in task directions with a D-lever sampling curriculum which derives from an optimal coverage criterion which measures the log-volume of the ellipsoid spanned by the policy embeddings. They show in 4 ExoRL environments that D-lever sampling is (somewhat) superior to uniform sampling when used in the Forward-backward (FB) and TD-JEPA algorithms and that it facilitates transfer from an environment to another in the low data regime.
Critical strengths
- the formalization of the proposed sampling criterion is clean and interesting
Critical weaknesses
** The experimental study suffers from several weaknesses:**
- the authors compare D-lever to A-lever, E-lever and APS-exact, but A-lever and E-lever are not described anywhere and APS-Exact is hardly described in appendices
- Figure 1 does not have any variability information, given the small differences in performance it is hard to conclude that these differences are statistically significant
- Figure 2(a) shows unstable performance under both conditions on Hopper, Figure 2(b) shows more steady progress but no gain from D-lever. The fact that D-lever sampling outperforms uniform sampling in Hopper is questionable, so it is hard to conclude from Fig. 2 that sampling from D-lever brings something. In that respect, Table 1 is more convincing.
- In 5.4, parag "Coverage expansion under sparse rewards", the main argument of the parag. relies on Table 5, which is in an Appendix, whereas it is unclear what Figure 3 brings.
Questions for authors
- As D-lever sampling follows a curriculum, it seems more appropriate to leverage it in an online learning than in an offline learning context. On the other hand, FB is offline. Could better explain how D-lever is leveraged in the online and the offline setting? My feeling is that this point deserves being put more forward.
- Could you explain A-lever and E-lever, please?
Minor remark: this paper looks closely related to [1], which shares the same sampling concern, but in a purely offline context. A comment on that is welcome.
[1] Nazim Bendib, Nicolas Perrin-Gilbert, and Olivier Sigaud. Improving zero-shot offline RL via behavioral task sampling, ICML 2026.
3 comments
Authors' Response (2/3)
Experimental results and interpretation
We thank the reviewer for the comments on our experimental evidence. We report variability across seeds in RiverSwim, provide a task-level breakdown of the Hopper results and discuss the contrast with Ant, and clarify what the Maze trajectories reveal about the curriculum.
Variability in RiverSwim
W2. Figure 1 does not have any variability information, given the small differences in performance it is hard to conclude that these differences are statistically significant.
We agree that variability information should be shown. Figure 1 is based on five seeds, and the variability across seeds is over 10× smaller than the gaps between methods. For example, at the final training step, the mean optimality ratio is for D-LEVER and for Uniform (mean standard error across seeds).
Since a single step does not reflect the whole learning curve, we also compare methods by the area under the curve, i.e., each metric averaged over all 21 evaluation steps of training, reported as mean standard error across the five seeds.
| Metric | D-LEVER | A-LEVER | E-LEVER | Uniform | APS |
|---|---|---|---|---|---|
| Mean | 0.899 ± 0.001 | 0.881 ± 0.001 | 0.870 ± 0.002 | 0.888 ± 0.001 | 0.882 ± 0.001 |
| CVaR$_{10}$ | 0.796 ± 0.003 | 0.754 ± 0.003 | 0.718 ± 0.004 | 0.766 ± 0.006 | 0.767 ± 0.002 |
| 25.09 ± 0.16 | 24.50 ± 0.22 | 23.63 ± 0.35 | 23.13 ± 0.19 | 23.20 ± 0.26 |
D-LEVER achieves the best value on every metric, with gaps that are large relative to the variability across seeds. The gap is smallest for coverage against A-LEVER, which optimizes a closely related criterion. We will add standard-error bands to Figure 1 and report the full statistical tests, including final-step comparisons, in the appendix.
Gains in Hopper task
W3. Figure 2(a) shows unstable performance under both conditions on Hopper, Figure 2(b) shows more steady progress but no gain from D-lever. The fact that D-lever sampling outperforms uniform sampling in Hopper is questionable, so it is hard to conclude from Fig. 2 that sampling from D-lever brings something. In that respect, Table 1 is more convincing.
We agree that the aggregate curves in Figure 2 alone are not conclusive. They average over all preferences, which obscures where D-LEVER helps. To make this clearer, we split the Hopper evaluation preferences into two groups according to which objective receives the larger weight, forward velocity or height . For each group, the table reports the return of the final policies and the early termination rate, i.e., the fraction of episodes that end before the time limit because the agent falls, as mean ± standard error across seeds.
| Hopper, final | Uniform () | D-LEVER () | Uniform () | D-LEVER () |
|---|---|---|---|---|
| Return | 243.6 ± 16.0 | 238.8 ± 20.2 | 205.6 ± 39.6 | 270.3 ± 8.4 |
| Early termination (%) | 20.7 ± 10.5 | 19.1 ± 6.9 | 38.5 ± 14.6 | 13.5 ± 6.6 |
The two methods are similar on forward-velocity-dominant tasks. On height-dominant tasks, however, uniform sampling often terminates early and varies widely across seeds, whereas D-LEVER performs reliably. D-LEVER's gain thus comes from behaviors that uniform sampling under-serves, which is what the coverage criterion targets. On Ant, both objectives concern planar locomotion and uniform sampling already serves them comparably, so no gain is expected, as discussed in Section 5.3. We will add this breakdown to the revision, along with additional seeds.
Visualization under sparse rewards (Figure 3)
W4. In 5.4 parag "Coverage expansion under sparse rewards", the main argument of the parag. relies on Table 5, which is in an Appendix, whereas it is unclear what Figure 3 brings.
We agree that the role of Figure 3 should be clearer and will revise the paragraph accordingly. Figure 3 is meant to show that, although sparse returns may show no progress, the curriculum nonetheless operates as intended.
In Maze, episodes start from the upper left, so goals farther from the start are inevitably less covered. Although the bottom-right goal still yields zero return (Table 5), the leverage scores in Figure 3 nonetheless reflect this ordering, rising from for the top-left policy to for the bottom-right one, so D-LEVER directs more training there.
While turning this behavioral coverage into reliable goal-reaching success remains a direction for future work, we are working to further support this observation with quantitative results from training and hope to share them within the discussion period.
Authors' Response (3/3)
Related works
Minor remark. this paper looks closely related to [1], which shares the same sampling concern, but in a purely offline context. A comment on that is welcome.
We thank the reviewer for raising this comparison, which helps clarify the position of our work. Both works share the motivation that uniform task sampling is suboptimal, but they pull in nearly opposite directions. Bendib et al. [1] use the offline dataset to identify task directions corresponding to data-supported behaviors and concentrate training on those directions. D-LEVER uses the current policy family as its reference and prioritizes behaviors that remain under-covered. The distinction is therefore between adapting training to the behaviors represented in the data and adapting it to the coverage of the policies being learned. This also explains D-LEVER's applicability to online learning, where the same curriculum can guide data collection to provide additional support for under-covered behaviors.
We also attempted to include [1] as a baseline. As no public implementation was available, we reimplemented the method from the paper. However, we could not reproduce the reported performance despite our efforts, and therefore did not include a comparison based on that implementation. We would be glad to revisit this comparison once a reference implementation becomes available.
These approaches could also be combined. The task distribution learned in [1] could provide a more informative base distribution for D-LEVER by reflecting the behaviors supported by the data. D-LEVER would then adaptively reweight this distribution according to the current policy family's coverage. We will add this discussion to the revised manuscript.
[1] Nazim Bendib, Nicolas Perrin-Gilbert, and Olivier Sigaud. Improving zero-shot offline RL via behavioral task sampling, ICML 2026.
Authors' Response (1/3)
We are sincerely grateful for the reviewer’s constructive comments and for recognizing the value of our formulation of the sampling criterion. We clarify how the curriculum operates in online and offline learning, explain the comparison methods, and address the experimental results and related work below. We will incorporate these clarifications into the revised manuscript by the end of the discussion period.
D-LEVER in online and offline settings
D-LEVER is a curriculum over training task directions that allocates more learning effort to latents whose policies contribute to under-covered directions in policy-embedding space. A fixed dataset still allows this allocation to change, since the same transitions can be used to train policies conditioned on different task directions.
In both online and offline learning, we therefore use D-LEVER to sample the latents used for training updates. The sampling distribution is periodically refreshed as the policies and representations evolve, so the curriculum responds to the coverage of the current policy family even when the data are fixed.
In the online setting, we also sample latents from the same curriculum distribution and execute the corresponding policies to collect new transitions. This directs data collection toward under-covered behaviors, providing more data support for the policies that the curriculum prioritizes for improvement. Both settings therefore allocate learning effort according to behavioral coverage, while online interaction additionally enriches the data available for that learning. We will clarify this connection in the main text.
Description of the comparison methods
We thank the reviewer for pointing out that these comparisons need a clearer explanation. Full definitions and implementation details are given in Appendix A.3.1, and to make the comparison easier to follow, we summarize the key ideas here. We will also bring this explanation into the main text.
A-LEVER and E-LEVER share the candidate pool, learner, and sampling framework of D-LEVER, and differ only in the coverage criterion. Classical optimal experimental design offers several ways to measure how well a set of embeddings covers the space, each corresponding to a different objective on the regularized second-moment matrix G of the policy embeddings:
\text{A: }\min\\,\operatorname{tr}(G^{-1}),\qquad \text{E: }\max\\,\lambda_{\min}(G).$$ D-optimality enlarges the overall volume spanned by the embeddings, A-optimality reduces the average uncertainty across directions, and E-optimality targets the single least-covered direction. Each method scores a candidate embedding by how much adding it improves its own objective. Comparing them tests whether the gains come specifically from D-optimality rather than from coverage-based allocation in general. APS-Exact is our tabular adaptation of APS [1] for RiverSwim. Whereas the D-LEVER variants keep each task's reward and only change which tasks are trained, APS changes the reward itself so that policies jointly visit diverse states: $$\text{APS: }\max\\;\underbrace{\mathbb{E}_z\big[\text{return on task } z\big]}_{\text{task}}+\underbrace{\mathcal{H}(\bar d)}_{\text{state diversity}},$$ where $\bar d$ is the state occupancy averaged over a batch of tasks. Original APS estimates the entropy with $k$-nearest neighbors over sampled states; since occupancies in RiverSwim can be computed exactly, APS-Exact uses the exact entropy instead. --- [1] Liu and Abbeel. APS: Active pretraining with successor features. International Conference on Machine Learning (ICML), 2021.