acceptodds
Under review as a conference paper at ICLR 2027

Not All Transitions Are Equal: Uncertainty-Driven Contrastive Learning for Visual RL

Abstract

Recent works show that prioritizing transitions from the replay buffer can improve representation learning by focusing on informative experiences, potentially guiding exploration and improving sample efficiency. We present UCRL: **U**ncertainty-Driven **C**ontrastive Learning for Visual **R**einforcement **L**earning, a method that leverages transition uncertainty to prioritize experiences for contrastive representation learning. Unlike existing contrastive RL methods, which uniformly sample transitions for representation learning, UCRL modifies only the transition sampling distribution of the contrastive objective. It prioritizes transitions according to epistemic uncertainty estimated from an ensemble of critics. We theoretically establish that nonzero epistemic uncertainty lower-bounds the encoder gradient induced by the critic objective, identifying epistemic uncertainty as an observable proxy for representation learning potential. We apply UCRL to CURL and evaluate the resulting method on the DeepMind Control Suite 100K and Atari 100K benchmarks, achieving relative performance gains of 9.5% and 14.8%, respectively, over CURL. Across five of the six main DeepMind Control Suite tasks, UCRL reaches CURL's 100k performance using 46.4% fewer environment interactions on average, with reductions of up to 65% across individual tasks. Code will be available upon acceptance.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Representation Learning, Sample Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_T4n2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UCRL, an uncertainty-driven sampling method for contrastive learning in visual reinforcement learning. It estimates epistemic uncertainty using the prediction variance of multiple critics sharing a visual encoder and uses this estimate to prioritize contrastive learning batches, while preserving the underlying RL branch’s original sampling strategy. The method is instantiated on CURL and evaluated on DMControl 100K and Atari 100K.

Critical strengths
  • Clear research question and simple method. The paper addresses which experiences should receive representation-learning updates, a direction complementary to designing new auxiliary losses. Using separate sampling distributions for the RL and contrastive learning branches is a clear and practically useful idea.

  • Performance improvements across multiple tasks. UCRL-C achieves higher mean returns on five of the six main DMControl tasks and outperforms CURL on 17 of 26 Atari games. The Atari IQM human-normalized score increases from 11.72% to 15.45%, suggesting that the benefits are not confined to a single continuous-control task.

Critical weaknesses
  • W1. Gap between the uncertainty estimator and its theoretical interpretation. Equation (7) distinguishes value uncertainty from representation uncertainty, but the implemented estimator only measures critic disagreement conditioned on a fixed shared encoder. It therefore cannot distinguish inadequate representations from insufficiently fitted critics; a nonzero encoder gradient does not resolve this ambiguity. Moreover, Assumption 2 requires the sum of gradient cross-terms to be nonnegative for each sample. The positive sample average reported in Appendix H.1 is insufficient to verify this condition, and the fraction of samples violating it should be reported.

  • W2. Missing a strictly controlled sampling comparison. The implementation simultaneously changes the number of critics, batch sizes, and whether batches are shared between branches. However, Table 4 lacks a control that holds these settings and gradient flow fixed while changing only the contrastive branch to uniform sampling, preventing clear attribution of the gains to uncertainty-based sampling. The stop-gradient ablation does not fully eliminate these confounds. The absence of per-task results and uncertainty estimates for the ablations also limits assessment of the consistency of the gains.

Questions for authors

Q1. Could the authors add a strictly matched uniform-sampling baseline? Please keep the number of critics, gradient flow, dual-batch configuration, update frequency, and training budget identical, changing only the contrastive sampling strategy, and report per-task results across multiple random seeds.

Q2. Why is prioritization applied exclusively to the contrastive branch? Could the authors compare prioritization of the RL branch only, the contrastive branch only, and both branches under otherwise identical settings to establish the necessity of this design choice?

Reviewer_NLteOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UCRL: Uncertainty Driven Contrastive Learning for Visual Reinforcement Learning. The proposed algorithm utilizes an ensemble of critics to estimate epistemic uncertainty and provide prioritization in contrastive representation learning for visual reinforcement learning tasks. The algorithm keeps the sampling strategy used by the base RL algorithm, and uses its own sampling strategy for contrastive learning. While the reported performance gains are limited, the paper claims improved sample efficiency and computational time compared to the standard CURL algorithm.

Critical strengths
  • The use of prioritization is common in reinforcement learning. The paper brings the prioritization to contrastive learning for RL. The paper contributes on both theoretical and empirical aspects of the algorithm.
  • As reported, the algorithm shows significant improvement in sample efficiency and reduction in computational time. (See weaknesses and questions on this topic).
Critical weaknesses
  • Experimental Evaluation: The performance improvement is limited. Most of the performance improvements over CURL are within the noise. Table 8 does not report SEM/SD. The atari100K metrics such as relative gain or normalized scores are typically dominated by few games (Kangaroo and Kung Fu Master in this case). The gains reported on the 16 task DMC set seems much lower compared to the smaller subset reported in main paper,
  • No learning curves are provided. To provide a clear picture on sample efficiency claims, they must be provided.
  • Assumption 2: It is not clear why the assumption 2 is a valid assumption, the empirical validation provided in appendix averages over (s,a) hence is not a good validation as assumption is made for all (s,a).
  • Name of the algorithm: I strongly request the authors to change the name of their algorithm from UCRL to something else. UCRL (Upper Confidence Reinforcement Learning) and its many variants are (UCRL2,KL-UCRL, H-UCRL and many more) well-established uncertainty estimate based RL algorithms.
Questions for authors
  • Assumption 2 is defined for all (s,a). The empirical validation provided in Figure 14 averages across (s,a) so it is not a good validation. Can the authors provide a correct validation for the assumption? I am not certain about assumption's validity.
  • Please provide learning curves and not only final values. Please provide standard deviation or the standard error in Table 8. Does table 7 include standard deviation or the standard error?
  • Can the author also provide compute time of the proposed algorithm with 2x 512 batch size? It will help clarify the differences in the computational time due to algorithmic addition vs batch size.
  • I strongly request authors to change the algorithm's name for the reasons highlighted above. Since the contribution is more about the contrastive representation learning part rather than the reinforcement learning part, maybe UDCL: Uncertainty Driven Contrastive Learning is a possible choice.
Reviewer_u28QOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents a method to use transition uncertainty to prioritize experiences for contrastive representation learning. Specifically, it uses an ensemble of critics to extract epistemic uncertainty. They present some theoretical results to justify epistemic uncertainty as an observable proxy for representation learning potential. The method is evaluated through simulations of robot learning tasks (on standard benchmarks) where the proposed methods performs comparably with baseline but with much fewer samples.

Critical strengths

The two main ideas that this paper touches upon are both interesting and relevant for the community. The first, of extracting epistemic uncertainty for use in better robot learning can be a strong part of solution that addresses the data scarcity challenge in robot learning. The second, of finding which experiences to prioritize has potential benefits in data curation for robot learning beyond the specific use case in this paper. As such, I believe the paper has sound ideas, even if the components themselves are straightforward.

Critical weaknesses

However, the present form of this paper has a number of limitations. First, the empirical results only show a modest improvement over existing methods. Second, the theory is somewhat inconsistent with the actual proposed method and unclear how useful the bounds actually are. At this point, it feels like the results are preliminary but need a deeper investigation. The gains themselves over baselines are modest. While it seems that the proposed methods shows improvement over CURL, the comparison with stronger methods is less convincing

Questions for authors

Priorities are updated only for transitions that get sampled, so it would seem most of the buffer carries older priorities from older critics. Is that so? If that's the case, isn't it problematic and prioritizing the wrong things

The performance improvements in Tables 1 and 2 are bimodal. Some tasks the proposed method does well, but some others (e.g., Cheetah run) it performs much worse than baselines. It’s a missed opportunity not digging into this. What causes the failures? Is it just run of the luck with random seeds (in which case, the improvements would also be biased by that) or is there something systematic about these cases that degrade the performance. IF so, then that suggests a place to irmpove the method. I feel like that’s the kind of analysis one would expect from this paper.

Higher-uncertainty experiences may simply be rare or new transitions. I feel like there must be a deeper dive of which transitions get selected and why they get selected beyond just baseline comparison? Perhaps one can analyze this with some hand-crafted examples to better understand why the method works.

How does UCRL plus augmentation compare against CURL plus augmentation?

I think a number of natural baselines are missing. For example, randomized selection, prioritizing by recency, prioritizing directly by constrainstive loss, etc.

Were the baseline methods actually run afresh or were their numbers used from the papers? If they were not run afresh, is there a difference (for example) in the seeds used, in the hyper parameters, or the platforms use that make the comparison unfair?

How well were the baselines tuned compared to the effort of tuning the proposed methods? Table 2 shows significant variances in DrQ-v2’s scores.

Why does the theory bound the critic loss’s encoder gradient, but the sampling choice is applied to the contrastive loss, which doesn’t involve the critics? How is the bound useful in that case?

If all critics share an encoder, is the uncertainty estimate from the ensemble biased? Is that bias accounted for in constructing the epistemic uncertainty?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.