acceptodds
Under review as a conference paper at ICLR 2027

Control-centric Representation Learning using Action-free Datasets with Distinct Policies

Abstract

Action- and reward-free data are available in settings such as robot manipulation, video game play, and human egocentric recordings. Prior work uses these data to learn visual representations for downstream control with limited action-labeled data. However, time-correlated visual distractions can degrade the representations learned by these methods. Critically, in many common visual control tasks, a naive encoder cannot distinguish noise from controllable features, leading to poor generalization. In this work, we consider action-free trajectories grouped by known data-collection policy within a shared environment. We introduce CARDPol, a representation-learning objective that predicts policy identity from pairs of observations. Under appropriate assumptions, we show that the ground-truth endogenous representation minimizes our objective. Empirically, our proposed loss learns representations that exhibit reduced sensitivity to time-correlated exogenous noise. CARDPol improves distracted LIBERO success by 3–11 percentage points and DMC video-distraction returns by 25–53% over the strongest evaluated baseline in each setting. On human egocentric video from EgoDex, it achieves the lowest mean wrist-position decoding error, with the closest baseline exhibiting 15% worse error. Our results demonstrate that policy identity can provide useful supervision for learning control-relevant representations from high-dimensional observations without action labels.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning for control, reinforcement learning, semi-supervised learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6ArVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This study considers learning visual representations for downstream control from action-free video data. However, temporally correlated visual distractors can degrade the representations learned by these methods, leading to poor generalization. The paper introduces CARDPol, a representation-learning objective that predicts policy identity from pairs of observations. Under appropriate assumptions, the paper shows that the ground-truth endogenous representation minimizes this objective. CARDPol is evaluated on downstream tasks (DMC, LIBERO) and pose decoding on EgoDex.

Critical strengths
  1. The paper has a clear motivation, and the method is simple, intuitive, and easy to implement, requiring only a shared encoder and a policy-identity classifier.

  2. Compared with CRAFT, the work that directly inspired this paper, this work makes some theoretical improvements. CARDPol relaxes CRAFT’s relatively restrictive assumptions, enabling the framework to be applied to continuous control settings.

Critical weaknesses
  1. The theoretical assumptions limit the applicability of the method. CARDPol assumes that exogenous noise is stationary, but this condition is often difficult to satisfy in practice. For example, videos collected by different people in different rooms or under different lighting conditions often fail to satisfy this assumption. Moreover, even if the exogenous noise is stationary, could subtracting the population mean from each sample remove most of this stationary noise? Furthermore, Theorem 1 guarantees only the sufficiency of endogenous-state information, rather than invariance to exogenous information, so further evidence of policy robustness is still needed.
  2. The existing experimental results do not appear to establish the necessity of some of CARDPol’s design choices. For example, the K = 0 setting also achieves comparable performance, leaving the advantages of multi-step inverse dynamics unclear. Most importantly, when exogenous backgrounds correlate with policy identity, the classifier can identify the policy solely from background information, causing the encoder to learn features that should have been suppressed. Although CRAFT directly inspired this work, the paper provides no comparison with CRAFT’s results. A comparison in settings where both methods are applicable would help clarify CARDPol’s contribution.
Questions for authors
  1. Most of CARDPol’s evaluations satisfy the assumption of stationary exogenous noise. However, how applicable is this assumption to the common case of videos collected in different rooms or under different lighting conditions?
  2. Under the assumption of stationary exogenous noise, what advantages does CARDPol offer over simple mean normalization?
  3. Given that K = 0 already achieves comparable performance, what additional benefits can the multi-step inverse dynamics introduced in the paper provide, and what advantages does it offer for downstream control?
  4. Can the authors compare CARDPol with CRAFT in a setting where both methods are applicable to clarify CARDPol’s advantages?
  5. Can the authors directly evaluate robustness when backgrounds correlate with policy identity? In this setting, would the encoder learn exogenous features that should have been suppressed?

I would consider increasing my score if the authors can convincingly address these concerns.

4 comments

(Question 1 and Question 5) how applicable is this assumption to the common case of videos collected in different rooms and policy correlated distractions

The question of applicability of our assumptions with respect to data collected in different rooms and under different conditions reduces to a question about whether the distracting conditions are correlated with the policies. For instance, in our video distraction settings in DCS, in addition to the background video we also vary the color of the agent itself throughout the trajectory. However, this distraction is truly exogenous and has no correlation to the policy.

In the case where the distraction has some correlation to the policy, it is possible that the objective could find a shortcut solution to the optimization and overfit to the correlations, if they are easier to identify than the endogenous state itself. To this end, we perform an experiment where different policies operate under different lighting conditions. Below, present state decoding results (where lower is better) under different policy-correlated lighting conditions. As the lighting differences get more extreme, the state decoding performance degrades significantly (0.08 to 0.89 in Cheetah and 0.27 to 0.89 in Walker). However, these conditions break our assumption of the Exogenous Block MDP and we do not make claims therein.

However, we maintain that exogenous block MDPs have been studied by prior work and have been demonstrated to be realistic models of many real world processes [Andaz 2024, Misra 2024, Islam 2023]

EnvironmentNo lighting differenceSmall differenceModerate differenceExtreme difference
Cheetah0.080800.987550.839060.89503
Walker0.276060.596490.933340.89867

(Questions 2) Advantages of CARDPol offer over simple mean normalization

With a mean normalization, the visual appearances of the distractions will still be present due to the residual differences between each individual distraction frame and their mean. The static distraction is only static throughout the episode, but is freshly sampled when the environment resets. Because of this, there is likely to be little difference between the results we have presented in the paper and this experiment.

(Question 3) Given that K = 0 already achieves comparable performance, what additional benefits can the multi-step inverse dynamics introduced in the paper provide

It is typical in DMControl environments to use framestack. However, it is likely the case that using framestacking stores policy information in a single observation and contain all the information necessary to predict the policy. We perform additional experiments without framestacking where we sweep k∈{0,1,2,3,4,5,6,7,8,9,10}k\in\{0,1,2,3,4,5,6,7,8,9,10\} and demonstrate that K>1 leads to the best state-decoding across all four settings. As you can see in the video distraction environments for Cheetah and Walker,  K=10 and K=6 lead to a 19.9% and 7.1% improvement in state decoding in Cheetah and Walker, respectively.

EnvironmentBackgroundBest KBest MSEReduction vs K=0
cheetahclean50.107371.8%
cheetahvideo100.1299919.9%
walkerclean100.3274916.8%
walkervideo60.368347.1%

(Question 4) CRAFT vs. CARDPol

We present a comparison between CARDPol and CRAFT in a distracted Gridworld setting with an “ego agent” and 3 distracting agents. The ego agent is controlled by either of 2 goal-conditioned policies and the 3 distracting agents follow a uniformly random policy. The policy identity for the ego agent is simply its goal (either top left of the grid or bottom right). The observations are in pixel space, but the underlying discrete endogenous state is simply the ego agent’s position in the grid. We instantiate four different grid sizes from 3x3 to 9x9.

We train CARDPol and CRAFT using a 64-dimensional IMPALA encoder, and 3,000 training updates. From the learned representations, we perform linear regression to the true underlying state as well as do k-means clustering to assign different latent representations to discrete engodenous states. We include a random baseline for comparison which assigns independent 64-dimensional Gaussian features to each observation.

Position R² (higher is better), mean ± sample standard deviation across three seeds.

GridCARDPolCRAFTRandom
3×30.947 ± 0.0190.946 ± 0.007−0.306 ± 0.067
5×50.875 ± 0.0070.875 ± 0.045−0.115 ± 0.014
7×70.897 ± 0.0040.848 ± 0.007−0.050 ± 0.003
9×90.923 ± 0.0050.912 ± 0.006−0.023 ± 0.005

Clustering accuracy (%, higher is better), mean ± sample standard deviation across three seeds.

GridCARDPolCRAFTRandom
3×377.1 ± 3.0%86.2 ± 6.5%11.2 ± 0.9%
5×581.2 ± 6.0%76.9 ± 6.6%4.3 ± 0.6%
7×761.4 ± 8.1%57.9 ± 1.1%2.1 ± 0.1%
9×954.4 ± 4.2%66.7 ± 6.1%1.4 ± 0.2%

Across all grid sizes under linear regression decoding, CARDPol marginally outperforms CRAFT, but not to a significant extent. Compared to the random features, both CARDPol and CRAFT perform similarly well. Under clustering, CARDPol and CRAFT perform similarly well compared random features, but their (i.e., CARDPol vs CRAFT) relative ranking varies across settings.

Together, these results suggest that CARDPol and CRAFT are comparable in a setting where CRAFT is valid.

References

  • Andaz et al. (2024). Learning an Inventory Control Policy with General Inventory Arrival Dynamics. ICLR Workshop on Generative Models for Decision Making.
  • Misra et al. (2024). Towards Principled Representation Learning from Videos for Reinforcement Learning. ICLR.
  • Islam et al. (2023). Principled Offline RL in the Presence of Rich Exogenous Information. ICML.
  • Lamb et al. (2023). Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models. Transactions on Machine Learning Research (TMLR).

Thank you for the additional experiments, but my two main concerns remain:

  1. The lighting experiment shows that state decoding degrades, but it remains unclear how much this degradation affects downstream control.

  2. Without frame stacking, K>0K>0 improves state decoding, but the results do not establish whether it improves downstream control in the paper’s original setting with frame stacking.

AuthorsOpenReview ↗

Thank you for your engagement

Hello Reviewer 6Arv,

Thank you for your continued engagement.

  1. How much [lighting] degradation affects downstream control.

We are currently running experiments to demonstrate control on the lighting setting and will share them when available. We expect that the control performance will degrade significantly, given the state-decoding results.

However, we would like to stress that we do not claim our method will work in this extreme setting where there is correlation between the background distraction and the policy itself. As a real world example, if a human is washing dishes, their behavior will not be predictable by the time of day they are performing the task. As an obvious counter-example, something like making a bed will be correlated with time of day (ie morning) and tasks such as these break our assumptions.

  1. Downstream Control results with frame-stacking

Below we demonstrate downstream performance of encoders trained with different K using 3-framestack during RL deployment in the Video distraction setting (two seeds for each run). We report cumulative reward and percentage improvement over K=0. We can see that K=5 is the best choice in video distraction setting over these values of K. This indicates that the multi-step feature of the loss is critical to obtain the best performance.

KCheetah videoImprovement vs K=0Walker videoImprovement vs K=0
0511.9 ± 50.9—607.4 ± 62.3—
3536.5 ± 48.3+4.8%642.5 ± 28.0+5.8%
5559.9 ± 27.1+9.4%683.8 ± 91.3+12.6%
10545.5 ± 17.3+6.6%645.5 ± 17.3*+6.3%
Reviewer_bgQcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CARDPol, a method for learning visual representations from trajectories with policy-group labels but no action labels. It predicts the identity of the policy that generated a trajectory from observations at two timesteps within that trajectory, aiming to capture control-relevant information. Under certain assumptions, the authors prove that optimal representations retain endogenous-state information. They evaluate the method on DMC, LIBERO, OGBench, and EgoDex through downstream control, state decoding, and pose regression.

Critical strengths

The paper proposes a simple representation-learning objective: identifying the policy that generated a trajectory from observations at two timesteps, using behavioral differences between policies to extract control-relevant information without per-step action annotations. A methodological strength is the conversion of trajectory-level group supervision into a directly optimizable classification objective. Under explicit assumptions, the authors also prove that optimizing this objective yields representations that retain endogenous-state information, providing theoretical support for the design.

Critical weaknesses
  1. The theory relies on groups of policies with sufficiently different behaviors. However, the LIBERO and EgoDex experiments use task labels, which may group together different behavioral patterns or be directly distinguishable through objects and scenes.

  2. Related to the concern above, the main control experiments construct action-free datasets by withholding existing action labels. In practice, action labels are often easier to obtain than policy-group labels. The paper does not sufficiently explain which scenarios, beyond using task labels, provide access to policy identities but not action labels.

Questions for authors
  1. How should we interpret CURL’s relatively strong state-decoding performance and robustness to background distractions in Figure 2 (second only to CARDPol) given that its success rate is only 0.8%–4.4% across the three distracted LIBERO suites? Does this suggest that the metrics shown in Figure 2 do not necessarily reflect the usefulness of the learned representations?

  2. Do CARDPol’s gains persist when the same pretrained encoder is used with different downstream policies, such as the existing Transformer-based BC policy and a Diffusion Policy?

2 comments

We thank the reviewer for their comments and thorough review of our work. We address concerns below.

settings “which may group together different behavioral patterns or be directly distinguishable through objects and scenes”

We agree with the insight that our classifier objective may be able to directly be optimized via unique objects or scenes or behavioral patterns. In Appendix C.2, we demonstrate sensitivity to these behavioral patterns in DMControl by pre-training on datasets containing 3 and 7 policies each. As we increase the number of policies and therefore overall behavioral overlap, performance generally increases (+6.7% in Cheetah-Clean, +10.1% in Cheetah-Static, +23.5% in Cheetah-Video, and 35.2% in Cheetah-Hard Video ).

Additionally, the EgoDex data used in our experiments include tasks like “Tying shoelaces”, “Unfolding t-shirt” and “unlocking a ziplock bag” along with 108 other tasks. Many of these demonstrations are distinguishable through their objects alone. Still, with the large scale and diversity of the data, we are able to learn representations that capture endogenous components of the video (i.e. ground truth human pose) better than all baselines.

action information is easier to obtain than policy/task information

While action information is often easy to obtain for robot demonstrations, prior work has demonstrated that it is still valuable to study robotics in the action-free setting [Lee 2024, Bu 2024]. Accordingly, we study both an action-limited robot setting (LIBERO) and a human video setting where there is no de facto notion of actions. In this work, we show that a diversity of policy/task labels are effective supervisory signals for CARDPol, such as policy performance, robot task label,  and human task label.

How should we interpret CURL’s relatively strong state-decoding performance?

The bar chart in Figure 2a that represents the distractions in the LIBERO experiments (where performance is 0.8 - 4.4%) is the Hard-Video setting. For endogenous recovery, CURL is approximately 2x worse than CARDPol and for exogenous sensitivity CURL is approximately 5.32x worse than CARDPol.

Do CARDPol’s gains persist when the same pretrained encoder is used with different downstream policies

In this work, we use three different methodologies to learn policies: 1) with TD3 (DMControl experiments; Table 1), 2) with IQL (OGBench; Table 11), and 3) with MSE-style Behavior cloning (LIBERO; Table 2). Each of these domains extracts policies in a different way, demonstrating how the benefits from the CARDPol loss during pre-training is agnostic to the downstream policy extraction method.

References

[1] Lee et al. (2024). Behavior Generation with Latent Actions. ICML. [2] Bu et al. (2025). Learning to Act Anywhere with Task-centric Latent Actions. RSS.

Reviewer_bgQcOpenReview ↗

Response to Authors

Thank you for your response. The issues I care about have not been fully clarified, such as whether the LIBERO and EgoDex task groupings satisfy the theory’s assumptions about behavioral differences, how to interpret the gap between state decoding and downstream control performance, and whether CARDPol’s gains persist across downstream policies when the pretrained encoders are held fixed. I therefore maintain my rating.

Reviewer_QGCdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes CARDPol, an action-free representation learning objective for settings where trajectories are grouped by the policy that collected them. A shared encoder and a classifier are trained to predict policy identity from a pair of observations k steps apart. The method builds on CRAFT (Levine et al., 2025) but drops its main restrictions: identical start states, discrete deterministic dynamics, and per-timestep encoders. The theory shows that in finite Ex-BMDPs with exo-free, sufficiently distinct policies, the ground-truth endogenous encoder minimizes the loss, and any minimizer contains enough information to recover the endogenous state. It does not show that exogenous information is removed.

Critical strengths
  1. The objective is simple and the DMC/OGBench results are strong. It is a plain classification loss that drops into any encoder, and it keeps CRAFT’s core idea while making it usable in continuous control.

  2. The evaluation goes beyond return. The PolCon control separates “having policy labels” from “this particular objective”. The paper also pairs state decoding with a matched-distractor variance metric and says clearly that good decoding alone does not show invariance.

Critical weaknesses
  1. I’m not convinced the method works for the reason the paper gives. The theory is about policies producing different transitions, but Table 8 shows K=0 (same frame-stack on both inputs) does just as well. So it looks like the encoder is mostly learning to tell checkpoints apart from single frames. If that’s right, the gain over PolCon is basically cross-entropy vs. SupCon on the same labels. PolCon even has the lower exo-variance ratio.

  2. The robotics and human-video evidence is thin. LIBERO distractors sit beside the image, most gaps are within the standard errors, and λ appears to be tuned on the eval setting. The EgoDex probe is evaluated on frames seen in pretraining. Policy-correlated scenes, the realistic case, are untested.

Questions for authors
  1. Could you add an end-to-end DrQ+BC result with no pretraining on DMC, and a generic pretrained encoder (e.g. a frozen DINOv2) as reference points?

  2. Can you test a case where distractors are partially correlated with the policy or task label? Overlaid rather than side-by-side distractors in LIBERO would also be more convincing.

4 comments
AuthorsOpenReview ↗

Response to Weaknesses and Questions

Thank you for the detailed review. We address your concerns below:

Table 8 shows K=0 (same frame-stack on both inputs) does just as well. So it looks like the encoder is mostly learning to tell checkpoints apart from single frames

It is typical in DMControl environments to use framestack. However, it is likely the case that using framestacking stores policy information in a single observation and contain all the information necessary to predict the policy. We perform additional experiments without framestacking where we sweep k∈{0,1,2,3,4,5,6,7,8,9,10}k\in\{0,1,2,3,4,5,6,7,8,9,10\} and demonstrate that K>1 leads to the best state-decoding across all four settings. As you can see in the video distraction environments for Cheetah and Walker,  K=10 and K=6 lead to a 19.9% and 7.1% improvement in state decoding in Cheetah and Walker, respectively.

EnvironmentBackgroundBest KBest MSEReduction vs K=0
cheetahclean50.107371.8%
cheetahvideo100.1299919.9%
walkerclean100.3274916.8%
walkervideo60.368347.1%

Performance of unfrozen and off-the-shelf pretrained encoders

We show performance of end-to-end DrQ+BC where 1) the encoder is unfrozen and 2)  the encoder is a frozen ResNet18 pre-trained on ImageNet. We opt for a pre-trained ResNet as opposed to a Dino-v2 model (as the reviewer suggests) because the ResNet18 has approximately the same number of parameters as our default encoder. We run 4 seeds of each in each of the 8 DMControl environments (2 domains x 4 distraction conditions). The pre-trained ResNet18 performs worse than our method on all conditions. The unfrozen encoder out-performs our frozen encoder in Cheetah Clean, Static, and Video, but underperforms in the other 5 settings. We keep the encoder frozen to evaluate the quality representations themselves, following prior work [Islam 2023, Misra 2024, Lamb 2023].

EnvironmentBackgroundCardPol (4 seeds)Frozen ResNet-18 (2 seeds)Unfrozen encoder (4 seeds)
CheetahClean544.6 ± 16.1364.6 ± 6.2688.5 ± 12.7
CheetahStatic622.1 ± 20.0335.8 ± 71.0663.2 ± 9.0
CheetahEasy video471.2 ± 24.7130.5 ± 9.3584.6 ± 41.4
CheetahHard video300.7 ± 66.4102.9 ± 10.1380.8 ± 21.8
WalkerClean895.7 ± 32.0565.7 ± 39.8845.1 ± 93.2
WalkerStatic625.9 ± 61.5196.8 ± 17.9412.5 ± 78.0
WalkerEasy video591.3 ± 28.0160.9 ± 3.7329.6 ± 21.4
WalkerHard video413.9 ± 16.0144.0 ± 8.1273.8 ± 48.0

Policy-correlated Distractions

Please refer to our response to Reviewer 6ArV for an answer to this question. In short, policy-correlated distractions break the assumptions of the Exogenous Block MDP and we demonstrate that this can be damaging to performance. However, we maintain that exogenous block MDPs have been studied by prior work and have been demonstrated to be realistic models of many real world processes [Andaz 2024, Misra 2024, Islam 2023]

Overlaid Distractions in LIBERO

We create a domain in LIBERO where the entire background is replaced by a distracting video (see anonymous video repo for example). We test our method against vanilla BC, VIP, VAE, and CURL. We determine auxiliary loss coefficients for each method independently by a small sweep over λ∈{0.001,0.01,0.1,1.0}\lambda \in \{0.001, 0.01, 0.1, 1.0\} and choose the top performing values for each method. In the table below, we show that CARDPol remains the top performing method in this new distraction setting. We additionally run more evaluation trajectories to get tighter error bounds.

BaselineBackground Distraction GoalBackground Distraction Object
No Pretraining57.10 ± 1.6940.25 ± 0.75
CardPol60.87 ± 0.5244.85 ± 4.95
VIP54.26 ± 1.9840.83 ± 2.06
CURL58.04 ± 1.3533.43 ± 0.66
VAE56.74 ± 1.5440.75 ± 0.55

The EgoDex probe is evaluated on frames seen in pretraining.

We pretrain the encoders on EgoDex videos without using their ground-truth pose labels. We then freeze each encoder and fit a supervised pose decoder on a smaller pose-labeled video dataset. We report pose prediction MSE on both the smaller dataset used to fit the decoders and EgoDex, whose pose labels are used only to evaluate the decoders. CARDPol achieves the lowest MSE on both datasets.

We present the MSE from the probes on both these data. We see the same relationship between the different methods, where CARDPol achieves the lowest error.

MethodTest MSE (×10⁻³) ↓Train MSE (×10⁻³) ↓
CARDPol3.384.25
VIP3.936.25
CURL4.054.96
VQ-VAE4.575.30
VAE3.896.57
ICVF5.566.43

References

[1] Andaz et al. (2024). Learning an Inventory Control Policy with General Inventory Arrival Dynamics. ICLR Workshop on Generative Models for Decision Making. [2] Misra et al. (2024). Towards Principled Representation Learning from Videos for Reinforcement Learning. ICLR. [3] Islam et al. (2023). Principled Offline RL in the Presence of Rich Exogenous Information. ICML. [4] Lamb et al. (2023). Guaranteed Discovery of Control-Endogenous Latent States with Multi-Step Inverse Models. Transactions on Machine Learning Research (TMLR).

Dear Reviewer QGCd,

We would like to include some additional control results that address your Weakness 1 regarding the importance of the Multi-step control.

Downstream Control results with frame-stacking

Below we demonstrate downstream performance of encoders trained with different K using 3-framestack during RL deployment in the Video distraction setting (two seeds for each run). We report cumulative reward and percentage improvement over K=0. We can see that K=5 is the best choice in video distraction setting over these values of K. This indicates that the multi-step feature of the loss is critical to obtain the best performance.

KCheetah videoImprovement vs K=0Walker videoImprovement vs K=0
0511.9 ± 50.9—607.4 ± 62.3—
3536.5 ± 48.3+4.8%642.5 ± 28.0+5.8%
5559.9 ± 27.1+9.4%683.8 ± 91.3+12.6%
10545.5 ± 17.3+6.6%645.5 ± 17.3*+6.3%

Please let us know if there is anything else to further assuage your concerns. Thank you!

Reviewer_QGCdOpenReview ↗

Response to Author

Thank you for the additional experiments. I'm still not convinced that learning temporal dynamic helps. Although the new results show some improvements with K > 0, the improvement are relatively modest compared to the reported variance. Therefore, I would like to maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.