- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors address the zero-shot RL (ZSRL) setting where all tasks have a shared permanent reward component. They claim that in this setting, learning a reward free latent representation and extracting a task-specific reward representation at inference is inefficient, as small approximation errors on the shared reward component can dominate the signal from the learned task-specific reward representation component. To mitigate this issue, they introduce the reward-aware Laplacian representation (ReAL), a modification of the standard Laplacian representation used in standard ZSRL research. They show experimentally that ReAL outperforms FB and the standard Laplacian method in a series of antmaze environment variants with a low reward region.
Critical strengths
- the authors clearly identify a limitation of the FB and standard Laplacian frameworks, which has to be addressed
Critical weaknesses
-
The paragraph "Choice of f" seems to imply that this function has to be carefully chosen depending on the properties of such or such shared reward function component: here it is defined for shared reward functions corresponding to low negative rewards. If this is correct, this is a strong limitation of the method: for instance, what if the shared reward component is positive, but lower than that of the goal? Do we have to define another f function?
-
the experimental study is focused on a very narrow set of environments that all share the same permanent reward component structure. It is not clear at all whether the proposed issue and mitigation extend to other domains than maze navigation with a permanent negative reward area. For instance, locomotion tasks (e.g. cheetah penalizing back-leg use) are missing.
-
I believe that the paragraph "Relationships with other representations" should be more forward in the introduction as it contains important elements for the positioning of the work. In particular, if the authors fail to extend their experimental study beyond their narrow ZESP instances, another positioning with respect to these related methods might be possible.
-
the comparison with the baselines is not clean: The FB baseline is not classic FB. Training z’s are replaced by (or ), renormalized, with periodically re-fit, and a single USFA is shared across all h (Appendix A.2.3). Since FB trains the encoder and policy jointly, this changes what the representation learns. There are signs that this handicaps the baselines: at the test reward is the standard goal reward, yet FB drops from 98.8% (Table 6) to 74.0% (Table 8) on antmaze-mn, and from 66.4% to 13.6% on antmaze-ls. It also means the ReAL − FB gap at (+24.0 on mn, +40.4 on ln) cannot come from reward-awareness, since no penalty exists there.
-
the mechanism and generality of the claim are not validated. The Gibbs-phenomenon explanation is an analogy backed only by indirect evidence. The approximation error of the permanent reward is never measured, and no theorem shows ReAL reduces it. The permanent reward is also a sharp discontinuity, exactly where Gibbs errors arise, so it is unclear what happens with a smooth penalty (e.g. an inverted Gaussian).
Questions for authors
-
In the maze setting of Fig. 1, are the results the same if the agent gets a large positive reward for reaching the goal, a penalty in the low reward region and 0 elsewhere? And if the low reward region corresponds to a positive reward lower than that of the goal?
-
to address these other settings, do you have to change the function or other parameters?
-
Success rate and permanent reward: Success only requires reaching the goal, so crossing the low-reward region still counts, and the return formula is never stated. Many baseline returns are near −1000 (failed episodes), so the return gap mostly reflects "reached the goal or not." Could you state how the return is computed, and report a success rate that only counts episodes reaching the goal without crossing the low-reward region?
-
Could you add FB (and the Laplacian) with standard z-sampling, and with a separate USFA per h, to show the gains are not due to the modified pipeline?
Minor issues, typos:
- Proposition 4.3 is trivial and could be presented as a remark.
- line 13: RL -> reinforcement learning (RL)
- Fig. 2 left, it is not clearly specified where are ground goal localizations (red areas ?) and which corresponds to which infered goal localization
7 comments
Rebuttal (2/2)
Q4: Could you add FB (and the Laplacian) with standard z-sampling, and with a separate USFA per h, to show the gains are not due to the modified pipeline?
A4: None of the requested baselines close the significant performance gap between FB/Laplacian and the ReAL representation, confirming that the gain of the ReAL representation does not come from the training paradigm. The success rates for the requested baselines are below.
The success rates for FB with standard z-sampling:
| h | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| mn | 98.0 | 85.6 | 64.4 | 52.4 | 40.4 |
| ms | 81.6 | 33.2 | 25.6 | 24.4 | 20.4 |
| ln | 82.4 | 26.8 | 15.6 | 6.8 | 5.6 |
| ls | 77.6 | 2.8 | 1.6 | 2.8 | 2.0 |
The success rates for FB where each is trained separately:
| h | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| mn | 97.6 | 94.8 | 74.4 | 34.4 | 38.4 |
| ms | 76.4 | 27.2 | 19.6 | 10.8 | 15.2 |
| ln | 76.4 | 6.8 | 1.6 | 1.2 | 0.4 |
| ls | 72.0 | 1.6 | 0.4 | 0.0 | 0.4 |
The success rates for Laplacian with standard z-sampling:
| h | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| mn | 98.4 | 76.4 | 59.6 | 44.4 | 33.2 |
| ms | 97.6 | 24.0 | 12.8 | 14.0 | 9.6 |
| ln | 90.4 | 29.2 | 16.0 | 14.8 | 9.2 |
| ls | 47.2 | 4.0 | 2.4 | 3.2 | 2.0 |
The success rates for Laplacian where each is trained separately:
| h | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| mn | 95.2 | 50.0 | 28.0 | 15.6 | 15.2 |
| ms | 98.0 | 28.0 | 20.4 | 15.6 | 8.4 |
| ln | 92.8 | 5.6 | 3.2 | 1.6 | 3.2 |
| ls | 50.8 | 3.6 | 4.8 | 4.0 | 4.4 |
Finally, the success rates of ReAL from the paper, for easy comparison here:
| h | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| mn | 98.0 | 97.6 | 98.4 | 97.2 | 97.2 |
| ms | 96.8 | 97.6 | 75.2 | 65.2 | 46.0 |
| ln | 87.6 | 76.8 | 64.4 | 49.6 | 39.2 |
| ls | 34.4 | 31.2 | 24.4 | 15.2 | 10.0 |
Q5: The permanent reward is also a sharp discontinuity, exactly where Gibbs errors arise, so it is unclear what happens with a smooth penalty (e.g. an inverted Gaussian).
A5: We further run a version of the experiments where the penalty is an inverted Gaussian multiplied by . Note that for FB and Laplacian, we choose the training paradigm (mentioned in Q4) that gives the highest success rates.
Success rate of OGBench with inverted Gaussian penalty, averaged over all four antmaze settings:
| h | 0 | 1 | 2 | 3 | 4 |
|---|---|---|---|---|---|
| Laplacian | 84.2 | 35.7 | 24.1 | 19.7 | 17.5 |
| FB | 87.1 | 37.0 | 24.7 | 20.3 | 17.8 |
| ReAL | 71.0 | 64.0 | 54.8 | 46.8 | 38.6 |
We can see that ReAL still significantly outperforms baselines when . We further note that the lower number when is possibly an artifact of the shared USFA training for different . Note that Laplacian and FB numbers here are trained with standard z-sampling, and thus do not suffer from this issue.
Q6: The mechanism and generality of the claim are not validated. The approximation error of the permanent reward is never measured, and no theorem shows ReAL reduces it.
A6: Measuring the approximation error of the permanent reward is not sufficient on its own. This is because approximation errors for just the permanent reward component could be cancelled or amplified by the approximation errors for the task specific component. Therefore, looking at how well each component is approximated in isolation does not inform us of whether task specific information is retained in the full approximated reward.
This is the reason we report metrics on the full approximated reward, and we have provided different forms of evidence validating the mechanism. Figure 1, 6, 7, and 8 are qualitative evidence that directly shows visually that local approximation errors in the fitted reward function (red peak with the black marker) overwhelm the goal peak (green marker). Figures 2 and 4 are quantitative evidence measuring whether the peak of fitted reward function lies near the actual goal.
Conclusion. We believe these results address the reviewer’s concerns about generality, the fairness of the baseline comparison and the mechanism, and we kindly ask the reviewer to consider raising their score.
Revised version?
The answers to my questions seem to all go in the right direction, but I would like to see the corresponding changes in the paper to make sure. Could the authors quickly share a revised version with the changes highlighted (e.g. in blue)? In particular, I want to make sure that the paper makes appropriate claims about the types of "common reward structures" that the algorithm satisfactorily deals with.
Rebuttal (1/2)
Summary. We thank the reviewer for the helpful feedback. We extended the experiments to humanoidmaze, visual antmaze and the DeepMind Control Suite (Appendix D), where ReAL largely outperforms the baselines, using the same f,c, and λ as in the existing antmaze experiments with no further tuning. Because the permanent reward is normalized, our algorithm can be directly applied without changing f even when the shared reward is positive. We stated how returns are computed and reported the number of successful episodes that avoid the low-reward region, where ReAL leads by a wide margin. We retrained FB and the Laplacian with standard z-sampling and with a separate USFA per h, and none of these variants closes the gap to ReAL. We also showed that ReAL’s advantage holds for a smooth inverted-Gaussian penalty.
A1: We further include experiment results in humanoidmaze, visual antmaze, and also Deepmind Control Suite in Appendix D.
For the extra domains in OGBench, apart from very challenging settings such as humanoidmaze-large and visual antmaze-ls, where all methods struggle to perform well, the ReAL representation largely outperforms the Laplacian and FB representations. For DMC, the ReAL representation achieves significantly higher returns compared to the Laplacian and FB representations when the permanent reward component is non-zero.
Our findings in these extra domains confirm our previous findings for antmaze, demonstrating that our results hold across a wide variety of settings.
A2:
Summary: For our experiments, we have not performed extensive tuning of f, and simply used the sigmoid function due to its robustness. Furthermore, we normalize the shared reward component to have zero mean and unit variance, so our existing procedure can be directly applied to the case where the shared reward is positive, but lower than that of the goal.
Details: In our paper, we simply use the sigmoid function as f for our OGBench experiments without exploring other choices. We chose sigmoid as it is bounded and robust to outliers. Though we did not extensively tune the choice of f, as shown in the paper, the ReAL representation significantly outperforms the baselines.
Furthermore, as detailed in Appendix A.1, we normalize the permanent reward component over the dataset to have zero mean and unit variance. This, again, is a design choice that enables the algorithm to be applied robustly to different permanent reward components, without being heavily affected by their respective offsets and scales. As a result, even if the shared reward component is positive but lower than that of the goal, we can simply apply the existing procedure and learn the reward-aware basis functions.
Finally, we directly used the same function f with the same parameters (λ=1, c=0.5) that we used for antmaze for running experiments in humanoidmaze, visual antmaze, and Deepmind Control Suite without any tuning. As shown in Appendix D of the revised paper, the ReAL representation still significantly outperforms the baselines.
In general, while, same as any hyperparameter, extensive tuning could possibly further improve performance, we do not expect the benefits offered by the ReAL representation to only manifest with extensive tuning of f.
A3: The example we used in Section 3.2 demonstrates how the downstream reward function is computed (near line 149). For the general case, at every time step, the reward is simply 0 for reaching the goal (upon which the episode terminates), −h−1 for being in the low-reward region, and −1 otherwise. Therefore, the return that we measure already captures the reward for reaching both the goal and low-reward regions. An agent not reaching the goal will accumulate negative reward in all of the 1000 time steps and receive a return ≤−1000. If two agents both reach the goal using the same number of steps, the one that reaches the goal without entering the low-reward region receives a higher return.
We generate 200 episodes for each downstream task in each of mn, ms, ln, ls, and report the number of successful episodes without crossing the low-reward region for ReAL, Laplacian, and FB. The max is 4000 episodes (200 episodes for each of the 20 tasks). ReAL has significantly more such successes: