- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the evaluation of representation proxies for model selection in reinforcement learning. The authors argue that rank correlation with downstream performance does not necessarily reflect a proxy's ability to select strong pretrained representations. They introduce NFoO, which is a selection-oriented metric, and evaluate seven proxies based on reward, action, temporal structure, and effective rank. Experiments across a lot of settings show that proxies can select near-optimal representations despite varying rank correlations. The authors also demonstrate the effectiveness of these proxies for early stopping during pretraining.
Critical strengths
-
The paper addresses a practical problem in RL, as evaluating pretrained representations through downstream finetuning can be computationally expensive. The distinction between ranking quality and model selection quality is well motivated.
-
The paper presents an extensive experimental study, covering 15 Atari games, 16 pretraining methods, and 2 downstream RL learners, with over 20,000 finetuning runs. The results provide useful empirical evidence on how different proxies behave in ranking and selection tasks.
Critical weaknesses
-
The methodological novelty seems somewhat limited. NFoO is mathematically equivalent to one minus top-k normalized regret, which has already been used in offline policy selection (Zhang and Jiang, 2021). While applying this metric to representation proxies is useful, the metric itself does not appear to be a new contribution.
-
The generalizability of the findings remains unclear. The experiments are limited to Atari with frozen encoders, where pretraining and finetuning are performed on the same game. Moreover, the results in Appendices D.2 and H suggest that selection quality can be sensitive to the candidate pool, which raises concerns about the conclusion that proxy choice matters little.
Zhang and Jiang, 2021. Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning. NeurIPS 2021.
Questions for authors
-
How stable are the proxy rankings and selection results across different pretraining seeds?
-
Appendix H studies candidate pool sensitivity by removing existing methods. Would the conclusions still hold when adding new pretraining methods?
5 comments
Rebuttal [Part 2 of 2]
Q1 (pretraining seeds)
We thank the reviewer for this question. We use one pretraining seed per (method, game), following Zhang et al. (2024), who use 10 finetuning seeds and no repeated pretraining. Our conclusions do not depend on any single pretraining run, since each method's score aggregates 15 encoders pretrained independently, one per game. Repeating pretraining would mean re-running the entire study, so we prioritized breadth instead: 15 games (up from 9), 16 methods, and a second learner. We agree that repeated pretraining would further strengthen the study, and we will state the single pretraining seed explicitly among the limitations.
Q2 (adding new methods)
We thank the reviewer for the question. We chose the pool to span as many pretraining objectives as our budget allowed, covering contrastive, self-predictive, and generative objectives at three encoder sizes; objectives outside these families remain untested. To approximate the effect of a different pool, Appendix H evaluates the proxies on every sub-pool of 6 to 15 of our 16 methods, each of which can be viewed as a new candidate pool. We will state the coverage of our pool as a limitation and list other pretraining families as a direction for future work.
[Fu et al., 2021] Benchmarks for deep off-policy evaluation
[Zhang & Jiang, 2021] Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning
[Zhang et al., 2024] Light-weight probing of unsupervised representations for reinforcement learning
We hope that our responses address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you. Thank you again for the review!
Dear Reviewer xxUw,
Since the Author/Reviewer discussion period concludes on Nov. 18, we would be happy to address any remaining concerns or questions.
Thanks!
Rebuttal [Part 1 of 2]
We thank the reviewer for carefully reviewing our paper. We greatly appreciate the feedback! Please see below our responses to the comments.
We have also updated the paper by qualifying our original claims in more detail and adding additional experiments in the appendices based on the feedback (updated text in red).
W1 (novelty of NFoO)
We thank the reviewer for the reference. NFoOk is one minus the top-k normalized regret from offline policy selection, and we build on this connection deliberately: Section 2 identifies offline policy selection as the closest setting, and Section 3.2 adopts its regret@k principle (Fu et al., 2021). We now cite Zhang & Jiang (2021) in Sections 2 and 3.2 and state the equivalence more explicitly. Our contribution is not the metric itself but its use for validating representation proxies. A proxy exists to choose a representation without finetuning every candidate, yet RL representation proxies have been validated almost exclusively by rank correlation (Zhang et al., 2024; Lee et al., 2023). Our results show the two metrics can disagree.
W2 (generalizability and pool sensitivity)
Scope. We agree, and our Discussion lists the same limits: settings beyond Atari, encoders updated during RL, and cross-task transfer. Our setting follows Zhang et al. (2024), whose findings we test with more proxies, methods, and learners. Lines 259-261 in our Experimental Setup explicitly mention that our results should be read within this scope.
Pool sensitivity and "proxy choice matters little." We thank the reviewer for carefully reviewing our claims and evidence! Our initial submission reported per-game results but drew its claims at the aggregate level. We now analyze per-game failure modes in more detail and state our claims separately at each level, to make the difference between them explicit. We’ve updated claims in the Abstract, Introduction, Results, and Discussion sections to reflect the following sentiment: among the proxies we consider, all select similarly well (and well above random) in aggregate, whereas within games, failures are proxy-specific.
In support of these claims, we also have extended our analysis in Appendix H: we recompute NFoO on all sub-pools of size 6 to 16 and report (1) the difference in NFoO between the sub-pool and the full pool of size 16, (2) the fraction of sub-pools where this difference is less than -0.1, and (3) the gap between best and worst proxies on each sub-pool. We find the following:
Selection quality is not a pool artifact. Every proxy’s NFoO is robust to moderate changes in the pool, under both learners and within individual games. N-step return is the only proxy robust in every setting.
The comparison between proxies is not a pool artifact. Across methods, proxies remain similar to one another (particularly when 2 selections are allowed, i.e. NFoO2). Within a game, they still remain substantially different.
Our recommendations on which proxies to use thus now reflect the setting in which selection is being performed: in aggregate (e.g. when choosing a pretraining method), using any of the probing-based proxies considered will suffice, but per-game, n-step return is the safer choice as it is the only proxy robust across all settings we consider.