- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UCRL, an uncertainty-driven sampling method for contrastive learning in visual reinforcement learning. It estimates epistemic uncertainty using the prediction variance of multiple critics sharing a visual encoder and uses this estimate to prioritize contrastive learning batches, while preserving the underlying RL branch’s original sampling strategy. The method is instantiated on CURL and evaluated on DMControl 100K and Atari 100K.
Critical strengths
-
Clear research question and simple method. The paper addresses which experiences should receive representation-learning updates, a direction complementary to designing new auxiliary losses. Using separate sampling distributions for the RL and contrastive learning branches is a clear and practically useful idea.
-
Performance improvements across multiple tasks. UCRL-C achieves higher mean returns on five of the six main DMControl tasks and outperforms CURL on 17 of 26 Atari games. The Atari IQM human-normalized score increases from 11.72% to 15.45%, suggesting that the benefits are not confined to a single continuous-control task.
Critical weaknesses
-
W1. Gap between the uncertainty estimator and its theoretical interpretation. Equation (7) distinguishes value uncertainty from representation uncertainty, but the implemented estimator only measures critic disagreement conditioned on a fixed shared encoder. It therefore cannot distinguish inadequate representations from insufficiently fitted critics; a nonzero encoder gradient does not resolve this ambiguity. Moreover, Assumption 2 requires the sum of gradient cross-terms to be nonnegative for each sample. The positive sample average reported in Appendix H.1 is insufficient to verify this condition, and the fraction of samples violating it should be reported.
-
W2. Missing a strictly controlled sampling comparison. The implementation simultaneously changes the number of critics, batch sizes, and whether batches are shared between branches. However, Table 4 lacks a control that holds these settings and gradient flow fixed while changing only the contrastive branch to uniform sampling, preventing clear attribution of the gains to uncertainty-based sampling. The stop-gradient ablation does not fully eliminate these confounds. The absence of per-task results and uncertainty estimates for the ablations also limits assessment of the consistency of the gains.
Questions for authors
Q1. Could the authors add a strictly matched uniform-sampling baseline? Please keep the number of critics, gradient flow, dual-batch configuration, update frequency, and training budget identical, changing only the contrastive sampling strategy, and report per-task results across multiple random seeds.
Q2. Why is prioritization applied exclusively to the contrastive branch? Could the authors compare prioritization of the RL branch only, the contrastive branch only, and both branches under otherwise identical settings to establish the necessity of this design choice?