- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
disclaimer: I also reviewed a previous version of this work. This version adds stronger baselines, manipulation experiments and clear details. I might be slightly biased.
This paper proposes VIBES, which replaces pixel reconstruction in model-based reinforcement learning with an adversarial objective that encourages representations to preserve reward and dynamics information while ignoring distractors. The method is evaluated on the Distracting Control Suite, Realistic ManiSkill and a high-dimensional swimmer task. The revision expands the baseline comparisons and manipulation experiments, and provides more detailed implementation and representation analyses.
Critical strengths
(S1: Well-motivated approach) Avoiding reconstruction of irrelevant observations is a clear motivation, and the adversarial formulation provides an interesting approach to learning bisimulation-based representations. The paper is easy to follow.
(S2: Broader empirical evaluation) The additional HRSSM, RePo and DreamerV3 comparisons strengthen the evaluation. The manipulation experiments broaden the evidence, while the swimmer results demonstrate applicability beyond image observations.
Critical weaknesses
(W1. Representation analysis does not isolate the proposed component) Return wise, Reacher-easy achieves 925 without the adversary, compared with 930 for proposed method VIBES. The representation analysis in Figure 4 nevertheless focuses on this task without comparing against the non-adversarial version. My question is therefore, which properties of the learned representation actually depend on the adversarial term? This is probably the main weakness in understanding the proposed method.
(W2 types of distraction) (not critical, this was one of my previous suggestions too). Background, camera and colour distractions are evaluated together. Understanding which distractions is the bisimulation especially good against would be a good additional contribution.
(W3. Adversarial contribution on ManiSkill) (not critical) The authors reasonably explain why the adversarial term contributes little on some DCS tasks. However, I think ManiSkill tasks can be configured to at least test the fully observable MDP hypothesis. A non-adversarial ablation would clarify how much the adversarial term contributes to ManiSkill.
Questions for authors
I would consider raising my score further if the questions are suitable responded. Q is for questions, C is for comments.
(Q1: Addresses W1) How do the representations differ with and without the adversary on Reacher-easy and on a task where the performance gain is larger? That is, could the authors compare VIBES and no adversarial objective on a common evaluation set, using Reacher-easy and one task with a larger return gain?
(Q2: Addresses W2) Which types of distraction benefit most from the adversarial term? Can separate background, camera and colour evaluations support the proposed explanation?
(Q3: Addresses W3) Does the adversarial term improve performance on the new manipulation tasks when compared with the same model trained without it?
(C1.) Please include DCS learning curves alongside the final returns in Table 1. These would help distinguish improvements in learning speed from improvements in final performance.
(C2) Related to C1, I think Figure 3 was never cross-referenced in the manuscript. Adding more learning curves would help.
2 comments
Thank you for the additional experiments on ManiSkill. I am satisfied with the responses. While the authors could not fully address W1, I think their commitment to addressing it is sufficient. I'm happy to maintain my score.
Response to Reviewer Ja8m
We thank the reviewer for the constructive review. In response, we have made several additions to the paper: we ran the no-adversary ablation on Realistic ManiSkill, added learning curves on the Distracting Control Suite alongside the final returns in Table 1, and are conducting a representation-level comparison with the no-adversary ablation, which will appear in the final version.
Our additions
1. No-adversary ablation on Realistic ManiSkill (addresses W3 and Q3). We ran the no-adversary ablation on both tasks (6 seeds) and added it to Table 2:
On Lift Cube, the difference is not significant (Welch’s t-test, p = 0.36). This is consistent with the hypothesis in Section 4.1: the adversarial term matters least when the task-relevant state is fully observable, and Realistic ManiSkill’s static backgrounds obscure the agent and objects far less than the moving distractors in DCS. On Turn Faucet, VIBES is significantly better (p = 0.03), and the difference lies mainly in consistency across seeds. VIBES seeds are tightly clustered, whereas no-adversary seeds vary widely: two reach returns comparable to VIBES (673 and 817), two reach intermediate returns (238 and 389), and two never learn the task (20.4 and 50.6, close to the random-action return of 35.7). This is compatible with the same hypothesis, which concerns only where training converges: even when the adversarial term contributes nothing at the optimum (as we hypothesize for fully observable environments), it supplies useful gradient early in training, when the encoder is still missing information (consistent with the large adversary–dynamics likelihood gap early in training, shown in the new Appendix C.7). Here, removing the adversary makes learning substantially less reliable across seeds.
2. Learning curves on the Distracting Control Suite (addresses C1 and C2). We have added learning curves for the DCS tasks in Appendix C.8. Figure 3 is now also cross-referenced in Section 4.3.
3. Planned representation analysis (addresses W1 and Q1). To further investigate the effect of the adversarial term, we plan a latent-space analysis on two Distracting Control Suite tasks: Reacher-easy, where the no-adversary ablation nearly matches VIBES’s final return (925 vs. 930), and Ball-in-cup-catch, where the two differ substantially (593 vs. 844). For each task, we will compare the latent spaces of both models using the analyses of Section 4.4, to isolate the effect of the adversary on the learned state representation. This requires more compute time than the discussion period allows, and the results will be included in the final version.