- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This study considers learning visual representations for downstream control from action-free video data. However, temporally correlated visual distractors can degrade the representations learned by these methods, leading to poor generalization. The paper introduces CARDPol, a representation-learning objective that predicts policy identity from pairs of observations. Under appropriate assumptions, the paper shows that the ground-truth endogenous representation minimizes this objective. CARDPol is evaluated on downstream tasks (DMC, LIBERO) and pose decoding on EgoDex.
Critical strengths
-
The paper has a clear motivation, and the method is simple, intuitive, and easy to implement, requiring only a shared encoder and a policy-identity classifier.
-
Compared with CRAFT, the work that directly inspired this paper, this work makes some theoretical improvements. CARDPol relaxes CRAFT’s relatively restrictive assumptions, enabling the framework to be applied to continuous control settings.
Critical weaknesses
- The theoretical assumptions limit the applicability of the method. CARDPol assumes that exogenous noise is stationary, but this condition is often difficult to satisfy in practice. For example, videos collected by different people in different rooms or under different lighting conditions often fail to satisfy this assumption. Moreover, even if the exogenous noise is stationary, could subtracting the population mean from each sample remove most of this stationary noise? Furthermore, Theorem 1 guarantees only the sufficiency of endogenous-state information, rather than invariance to exogenous information, so further evidence of policy robustness is still needed.
- The existing experimental results do not appear to establish the necessity of some of CARDPol’s design choices. For example, the K = 0 setting also achieves comparable performance, leaving the advantages of multi-step inverse dynamics unclear. Most importantly, when exogenous backgrounds correlate with policy identity, the classifier can identify the policy solely from background information, causing the encoder to learn features that should have been suppressed. Although CRAFT directly inspired this work, the paper provides no comparison with CRAFT’s results. A comparison in settings where both methods are applicable would help clarify CARDPol’s contribution.
Questions for authors
- Most of CARDPol’s evaluations satisfy the assumption of stationary exogenous noise. However, how applicable is this assumption to the common case of videos collected in different rooms or under different lighting conditions?
- Under the assumption of stationary exogenous noise, what advantages does CARDPol offer over simple mean normalization?
- Given that K = 0 already achieves comparable performance, what additional benefits can the multi-step inverse dynamics introduced in the paper provide, and what advantages does it offer for downstream control?
- Can the authors compare CARDPol with CRAFT in a setting where both methods are applicable to clarify CARDPol’s advantages?
- Can the authors directly evaluate robustness when backgrounds correlate with policy identity? In this setting, would the encoder learn exogenous features that should have been suppressed?
I would consider increasing my score if the authors can convincingly address these concerns.
4 comments
(Question 4) CRAFT vs. CARDPol
We present a comparison between CARDPol and CRAFT in a distracted Gridworld setting with an “ego agent” and 3 distracting agents. The ego agent is controlled by either of 2 goal-conditioned policies and the 3 distracting agents follow a uniformly random policy. The policy identity for the ego agent is simply its goal (either top left of the grid or bottom right). The observations are in pixel space, but the underlying discrete endogenous state is simply the ego agent’s position in the grid. We instantiate four different grid sizes from 3x3 to 9x9.
We train CARDPol and CRAFT using a 64-dimensional IMPALA encoder, and 3,000 training updates. From the learned representations, we perform linear regression to the true underlying state as well as do k-means clustering to assign different latent representations to discrete engodenous states. We include a random baseline for comparison which assigns independent 64-dimensional Gaussian features to each observation.
Position R² (higher is better), mean ± sample standard deviation across three seeds.
| Grid | CARDPol | CRAFT | Random |
|---|---|---|---|
| 3×3 | 0.947 ± 0.019 | 0.946 ± 0.007 | −0.306 ± 0.067 |
| 5×5 | 0.875 ± 0.007 | 0.875 ± 0.045 | −0.115 ± 0.014 |
| 7×7 | 0.897 ± 0.004 | 0.848 ± 0.007 | −0.050 ± 0.003 |
| 9×9 | 0.923 ± 0.005 | 0.912 ± 0.006 | −0.023 ± 0.005 |
Clustering accuracy (%, higher is better), mean ± sample standard deviation across three seeds.
| Grid | CARDPol | CRAFT | Random |
|---|---|---|---|
| 3×3 | 77.1 ± 3.0% | 86.2 ± 6.5% | 11.2 ± 0.9% |
| 5×5 | 81.2 ± 6.0% | 76.9 ± 6.6% | 4.3 ± 0.6% |
| 7×7 | 61.4 ± 8.1% | 57.9 ± 1.1% | 2.1 ± 0.1% |
| 9×9 | 54.4 ± 4.2% | 66.7 ± 6.1% | 1.4 ± 0.2% |
Across all grid sizes under linear regression decoding, CARDPol marginally outperforms CRAFT, but not to a significant extent. Compared to the random features, both CARDPol and CRAFT perform similarly well. Under clustering, CARDPol and CRAFT perform similarly well compared random features, but their (i.e., CARDPol vs CRAFT) relative ranking varies across settings.
Together, these results suggest that CARDPol and CRAFT are comparable in a setting where CRAFT is valid.
References
- Andaz et al. (2024). Learning an Inventory Control Policy with General Inventory Arrival Dynamics. ICLR Workshop on Generative Models for Decision Making.
- Misra et al. (2024). Towards Principled Representation Learning from Videos for Reinforcement Learning. ICLR.
- Islam et al. (2023). Principled Offline RL in the Presence of Rich Exogenous Information. ICML.
- Lamb et al. (2023). Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models. Transactions on Machine Learning Research (TMLR).
Thank you for the additional experiments, but my two main concerns remain:
-
The lighting experiment shows that state decoding degrades, but it remains unclear how much this degradation affects downstream control.
-
Without frame stacking, improves state decoding, but the results do not establish whether it improves downstream control in the paper’s original setting with frame stacking.
Thank you for your engagement
Hello Reviewer 6Arv,
Thank you for your continued engagement.
- How much [lighting] degradation affects downstream control.
We are currently running experiments to demonstrate control on the lighting setting and will share them when available. We expect that the control performance will degrade significantly, given the state-decoding results.
However, we would like to stress that we do not claim our method will work in this extreme setting where there is correlation between the background distraction and the policy itself. As a real world example, if a human is washing dishes, their behavior will not be predictable by the time of day they are performing the task. As an obvious counter-example, something like making a bed will be correlated with time of day (ie morning) and tasks such as these break our assumptions.
- Downstream Control results with frame-stacking
Below we demonstrate downstream performance of encoders trained with different K using 3-framestack during RL deployment in the Video distraction setting (two seeds for each run). We report cumulative reward and percentage improvement over K=0. We can see that K=5 is the best choice in video distraction setting over these values of K. This indicates that the multi-step feature of the loss is critical to obtain the best performance.
| K | Cheetah video | Improvement vs K=0 | Walker video | Improvement vs K=0 |
|---|---|---|---|---|
| 0 | 511.9 ± 50.9 | — | 607.4 ± 62.3 | — |
| 3 | 536.5 ± 48.3 | +4.8% | 642.5 ± 28.0 | +5.8% |
| 5 | 559.9 ± 27.1 | +9.4% | 683.8 ± 91.3 | +12.6% |
| 10 | 545.5 ± 17.3 | +6.6% | 645.5 ± 17.3* | +6.3% |
The question of applicability of our assumptions with respect to data collected in different rooms and under different conditions reduces to a question about whether the distracting conditions are correlated with the policies. For instance, in our video distraction settings in DCS, in addition to the background video we also vary the color of the agent itself throughout the trajectory. However, this distraction is truly exogenous and has no correlation to the policy.
In the case where the distraction has some correlation to the policy, it is possible that the objective could find a shortcut solution to the optimization and overfit to the correlations, if they are easier to identify than the endogenous state itself. To this end, we perform an experiment where different policies operate under different lighting conditions. Below, present state decoding results (where lower is better) under different policy-correlated lighting conditions. As the lighting differences get more extreme, the state decoding performance degrades significantly (0.08 to 0.89 in Cheetah and 0.27 to 0.89 in Walker). However, these conditions break our assumption of the Exogenous Block MDP and we do not make claims therein.
However, we maintain that exogenous block MDPs have been studied by prior work and have been demonstrated to be realistic models of many real world processes [Andaz 2024, Misra 2024, Islam 2023]
With a mean normalization, the visual appearances of the distractions will still be present due to the residual differences between each individual distraction frame and their mean. The static distraction is only static throughout the episode, but is freshly sampled when the environment resets. Because of this, there is likely to be little difference between the results we have presented in the paper and this experiment.
It is typical in DMControl environments to use framestack. However, it is likely the case that using framestacking stores policy information in a single observation and contain all the information necessary to predict the policy. We perform additional experiments without framestacking where we sweep k∈{0,1,2,3,4,5,6,7,8,9,10} and demonstrate that K>1 leads to the best state-decoding across all four settings. As you can see in the video distraction environments for Cheetah and Walker, K=10 and K=6 lead to a 19.9% and 7.1% improvement in state decoding in Cheetah and Walker, respectively.