acceptodds
Under review as a conference paper at ICLR 2027

From Ranking to Selection: Evaluating Representation Proxies in RL

Abstract

Pretraining representations can improve sample efficiency in reinforcement learning, but evaluating a representation normally requires expensive finetuning across many seeds and tasks. To avoid this cost, prior work scores representations with cheap proxies, such as linear probing against task-derived targets (e.g. reward) and direct measures of the representation itself (e.g. effective rank), and validates them by their rank correlation () with downstream performance. Such validation is incomplete, as scores the ordering of an entire pool of candidates, whereas the use case that matters in practice, model selection, depends only on the top choices. We therefore propose normalized fraction-of-oracle (), which scores a proxy by the downstream performance of the representation it selects relative to the best available, and argue it should be reported alongside . We also widen the pool of proxies under test: we evaluate a broad range of probing targets, spanning reward presence, multi-step return, behavioral cloning, episode structure, transition dynamics, and temporal structure. We assess seven proxies under both metrics in two model selection scenarios on the Atari benchmark: selecting among pretraining methods and among representations within a single game. We also evaluate their use for early stopping during pretraining. Across more than 20,000 finetuning runs on 15 games, we find an asymmetry: across proxies that capture different signals, proxy rankings shift with the candidate pool and downstream learner, yet their selections remain consistently strong. Representation proxies are therefore more robust as selectors than their rank correlations alone would suggest.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, representation learning, model selection

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_xxUwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the evaluation of representation proxies for model selection in reinforcement learning. The authors argue that rank correlation with downstream performance does not necessarily reflect a proxy's ability to select strong pretrained representations. They introduce NFoO, which is a selection-oriented metric, and evaluate seven proxies based on reward, action, temporal structure, and effective rank. Experiments across a lot of settings show that proxies can select near-optimal representations despite varying rank correlations. The authors also demonstrate the effectiveness of these proxies for early stopping during pretraining.

Critical strengths
  1. The paper addresses a practical problem in RL, as evaluating pretrained representations through downstream finetuning can be computationally expensive. The distinction between ranking quality and model selection quality is well motivated.

  2. The paper presents an extensive experimental study, covering 15 Atari games, 16 pretraining methods, and 2 downstream RL learners, with over 20,000 finetuning runs. The results provide useful empirical evidence on how different proxies behave in ranking and selection tasks.

Critical weaknesses
  1. The methodological novelty seems somewhat limited. NFoO is mathematically equivalent to one minus top-k normalized regret, which has already been used in offline policy selection (Zhang and Jiang, 2021). While applying this metric to representation proxies is useful, the metric itself does not appear to be a new contribution.

  2. The generalizability of the findings remains unclear. The experiments are limited to Atari with frozen encoders, where pretraining and finetuning are performed on the same game. Moreover, the results in Appendices D.2 and H suggest that selection quality can be sensitive to the candidate pool, which raises concerns about the conclusion that proxy choice matters little.

Zhang and Jiang, 2021. Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning. NeurIPS 2021.

Questions for authors
  1. How stable are the proxy rankings and selection results across different pretraining seeds?

  2. Appendix H studies candidate pool sensitivity by removing existing methods. Would the conclusions still hold when adding new pretraining methods?

5 comments
AuthorsOpenReview ↗

Rebuttal [Part 1 of 2]

We thank the reviewer for carefully reviewing our paper. We greatly appreciate the feedback! Please see below our responses to the comments.

We have also updated the paper by qualifying our original claims in more detail and adding additional experiments in the appendices based on the feedback (updated text in red).


W1 (novelty of NFoO)

We thank the reviewer for the reference. NFoOk\mathrm{NFoO}_k is one minus the top-k normalized regret from offline policy selection, and we build on this connection deliberately: Section 2 identifies offline policy selection as the closest setting, and Section 3.2 adopts its regret@k principle (Fu et al., 2021). We now cite Zhang & Jiang (2021) in Sections 2 and 3.2 and state the equivalence more explicitly. Our contribution is not the metric itself but its use for validating representation proxies. A proxy exists to choose a representation without finetuning every candidate, yet RL representation proxies have been validated almost exclusively by rank correlation (Zhang et al., 2024; Lee et al., 2023). Our results show the two metrics can disagree.


W2 (generalizability and pool sensitivity)

Scope. We agree, and our Discussion lists the same limits: settings beyond Atari, encoders updated during RL, and cross-task transfer. Our setting follows Zhang et al. (2024), whose findings we test with more proxies, methods, and learners. Lines 259-261 in our Experimental Setup explicitly mention that our results should be read within this scope.

Pool sensitivity and "proxy choice matters little." We thank the reviewer for carefully reviewing our claims and evidence! Our initial submission reported per-game results but drew its claims at the aggregate level. We now analyze per-game failure modes in more detail and state our claims separately at each level, to make the difference between them explicit. We’ve updated claims in the Abstract, Introduction, Results, and Discussion sections to reflect the following sentiment: among the proxies we consider, all select similarly well (and well above random) in aggregate, whereas within games, failures are proxy-specific.

In support of these claims, we also have extended our analysis in Appendix H: we recompute NFoO\mathrm{NFoO} on all sub-pools of size 6 to 16 and report (1) the difference in NFoO\mathrm{NFoO} between the sub-pool and the full pool of size 16, (2) the fraction of sub-pools where this difference is less than -0.1, and (3) the gap between best and worst proxies on each sub-pool. We find the following:

  • Selection quality is not a pool artifact. Every proxy’s NFoO\mathrm{NFoO} is robust to moderate changes in the pool, under both learners and within individual games. N-step return is the only proxy robust in every setting.

  • The comparison between proxies is not a pool artifact. Across methods, proxies remain similar to one another (particularly when 2 selections are allowed, i.e. NFoO2\mathrm{NFoO}_2). Within a game, they still remain substantially different.

Our recommendations on which proxies to use thus now reflect the setting in which selection is being performed: in aggregate (e.g. when choosing a pretraining method), using any of the probing-based proxies considered will suffice, but per-game, n-step return is the safer choice as it is the only proxy robust across all settings we consider.

AuthorsOpenReview ↗

Rebuttal [Part 2 of 2]

Q1 (pretraining seeds)

We thank the reviewer for this question. We use one pretraining seed per (method, game), following Zhang et al. (2024), who use 10 finetuning seeds and no repeated pretraining. Our conclusions do not depend on any single pretraining run, since each method's score aggregates 15 encoders pretrained independently, one per game. Repeating pretraining would mean re-running the entire study, so we prioritized breadth instead: 15 games (up from 9), 16 methods, and a second learner. We agree that repeated pretraining would further strengthen the study, and we will state the single pretraining seed explicitly among the limitations.

Q2 (adding new methods)

We thank the reviewer for the question. We chose the pool to span as many pretraining objectives as our budget allowed, covering contrastive, self-predictive, and generative objectives at three encoder sizes; objectives outside these families remain untested. To approximate the effect of a different pool, Appendix H evaluates the proxies on every sub-pool of 6 to 15 of our 16 methods, each of which can be viewed as a new candidate pool. We will state the coverage of our pool as a limitation and list other pretraining families as a direction for future work.


[Fu et al., 2021] Benchmarks for deep off-policy evaluation

[Zhang & Jiang, 2021] Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning

[Zhang et al., 2024] Light-weight probing of unsupervised representations for reinforcement learning


We hope that our responses address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you. Thank you again for the review!

Dear Reviewer xxUw,

Since the Author/Reviewer discussion period concludes on Nov. 18, we would be happy to address any remaining concerns or questions.

Thanks!

Reviewer_KrAyOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how cheap proxies for pretrained representation quality in RL should be validated. Prior work scores a proxy by its Spearman rank correlation ρ with downstream fine-tuning performance over a pool of candidate representations. The authors argue that ρ is misaligned with the practical model selection, which depends only on the top-ranked candidates. They propose normalized fraction-of-oracle (NFoO), the min-max normalized downstream score of the representation ranked top-k by a proxy, and recommend reporting it alongside ρ. Seven proxies are evaluated over 16 pretraining methods and 15 Atari games with frozen encoders, fine-tuned by Rainbow and PPO. Empirical evaluation finds that ρ varies widely across proxies and reorders across learners, whereas NFoO is consistently high. They conclude that proxies are more robust as selectors than ρ suggests and that the choice of proxy matters little for selection.

Critical strengths
  1. The paper is clean and easy to read. Numbers are reported compactly and informatively in the figures and tables. The appendix is substantial but organized, and the main body is well-equipped with pointers to the corresponding appendix sections, which makes verification easy.
  2. For an empirical study, the scale is acceptable: 16 pretraining methods spanning contrastive, self-predictive, and generative objectives, 15 games, two downstream learners, 10 seeds per run, and over 20,000 fine-tuning runs.
  3. Based on my own research experience, I agree with the paper’s evidence that effective rank is a poor proxy of representation quality for RL. Many recent RL papers accompany their performance improvements with an increased feature rank, yet I have seen plenty of cases where high feature rank gives low performance and vice versa, and recent studies [1, 2] suggest that task-relevant structure in RL and Imitation Learning can be low-dimensional, so rank is not a reliable indicator of feature quality. Section 5.1 actually reveals that it is unreliable, which agrees with my experience. I would likely cite this paper as evidence that effective rank is a poor proxy.

References:

[1] Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension (Andriopoulos et al., 2026)

[2] Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning (Todorov & Sabatelli, 2026)

Critical weaknesses

From my understanding, the main takeaways are: (1) when validating proxies, a low rank correlation should not disqualify a proxy, because ρ eclipses its selection quality and changes with the downstream learner, whereas the representation with the highest proxy score is reliably close to the best downstream performance (NFoO definition), at least within the paper’s scope of seven proxies and two learners; and (2) consequently, the choice of proxy matters little. I think both messages are undercut by the paper’s own results.

  1. A low rank correlation still disqualifies a proxy, and NFoO does not help much. Effective rank is the most chaotic proxy in the paper: it ranks representations with high rank but poor downstream performance near the top, and its NFoO is good only because those representations happened not to be ranked first. The authors acknowledge this is partly fortunate. In my experience, this is not an accident of this pool: high feature rank can coexist with low performance, so a proxy with low ρ is indeed unreliable, and a high NFoO on one pool risks being misleading. More generally, NFoO can only be computed once the downstream scores of the whole pool are known. A practitioner facing a new pool has no way to tell whether a low-ρ proxy will be lucky again and would reasonably reject it on ρ anyway. So I do not see what decision changes once NFoO is added.
  2. “The choice of proxy matters little” is too absolute. As an example, I would not choose effective rank given its low rank correlation, regardless of its luckily high NFoO. For the per-game results, is it true that: the games where a proxy fails differ from one proxy to another, and the proxy that works best differs from one game to another? If so, the choice of proxy depends on the game structure, which the medians over games hide. Across learners, the paper shows that the proxies with high rank correlation under Rainbow become the weakest under PPO and vice versa. Finally, Section 5.5 shows that outside two nested pairs, the proxies do not misrank the same representations, i.e., they capture different structure in the representation, yet reach similar top-1 performance. Taken together, these observations raise the question of whether different learners and tasks prefer different representational structure. If so, there is no universally good proxy, and selecting a proxy requires a downstream-relevant prior about the learner and the task, rather than “whichever proxy is computable” or “most convenient” as the paper advises. This is close to the opposite of takeaway (2).
  3. The practical and actionable insights of this study are unclear to me. Two concrete scenarios: (a) Suppose I want to study whether intrinsic dimension is a good proxy of representation quality. Beyond “report both ρ and NFoO,” how does this work help me along the way? (b) Suppose I want to choose a good pretrained representation for FlashSAC on HumanoidBench, or for PQN on Atari. What selection procedure does this work suggest? I am not ordering new experiments on these two scenarios; I would like to understand, at a high level, how the findings of this work concretely guide a researcher or a practitioner in such situations.
Questions for authors
  1. Regarding Weakness 1: in what situation would a practitioner act differently after seeing NFoO in addition to ρ, given that NFoO can only be computed on a pool whose downstream scores are known?
  2. Regarding Weakness 2: could the authors characterize when and why each proxy fails at the per-game level, and whether the learner type predicts which family of proxies is reliable? More evidence is needed to support the claim: “the choice of proxy matters little”.
  3. Regarding Weakness 3: could the authors advise the recommended procedure for the two scenarios above, and clarify what in this study transfers to a different learner or domain?
6 comments
AuthorsOpenReview ↗

Rebuttal [Part 1 of 3]

We thank the reviewer for carefully reviewing our paper. We greatly appreciate the feedback! Please see below our responses to the comments.

We have also updated the paper by qualifying our original claims in more detail and adding additional experiments in the appendices based on the feedback (updated text in red).


W1.1: “A low rank correlation still disqualifies a proxy, and NFoO does not help much.”

We agree that a low rho does imply the proxy orders the pool as a whole poorly. The lower the correlation, the easier it is to construct an adversarial pool composition on which selection fails. This is why we report ρ\rho and do not propose replacing it. We nevertheless argue that rho alone is insufficient to judge a proxy:

High rho does not imply good selection. On Demon Attack, the reward probe reaches ρ\rho = 0.72 over 100 representations (relatively high), yet its NFoO\mathrm{NFoO} for k=1,2,3k=1,2,3 is 0.18, 0.18, and 0.42, respectively. Moreover, its top pick falls below the random baseline of 0.27 (Tables 4-5). Judged by ρ\rho, this proxy is reliable on this game; used for selection (measured by NFoO\mathrm{NFoO}), it performs poorly. At least three other (proxy, game) cells have rho at least 0.7 but select below random.

ρ\rho is insensitive to magnitude. Consider 16 candidates ordered by downstream score, in two instances: in A the two best score 100 and 99, in B they score 100 and 50 (and say the worst is 0). A proxy that swaps the top two and orders the rest correctly has the same ρ\rho of 0.997 in both. Its pick loses 1% of the achievable range in A and 50% in B (NFoO\mathrm{NFoO} = 0.99 vs. 0.50). Rank correlation can’t separate a harmless error from a costly one.


W1.2/Q1: “NFoO can only be computed once the downstream scores of the whole pool are known…what decision changes?”

Like ρ\rho, NFoO\mathrm{NFoO} is a validation metric. Both metrics are computed on benchmark pools whose downstream scores are known. Regarding what decision changes with NFoO\mathrm{NFoO}, consider two examples from our experiments. By ρ\rho alone, one would trust the reward probe on Demon Attack (ρ\rho = 0.72), where it selects worse than random (see Tables 4-5). One may also lean towards rejecting the reward probe based under PPO (ρ\rho = 0.6; see Figure 3), where it picks the best method.

AuthorsOpenReview ↗

Rebuttal [Part 2 of 3]

W2.1: “I would not choose effective rank given its low rank correlation, regardless of its luckily high NFoO\mathrm{NFoO}."

Thank you for pointing this out! We have made the unreliability of effective rank more explicit now. We’ve updated our aside on effective rank in Section 5.1, noting:

  • Our observation that effective rank is biased towards ST-DIM corroborates recent work that implies that high feature rank alone does not indicate task-relevant features. We thank the reviewer for the provided references, and have included them in Section 5.1.

  • Appendix H (which we have extended from the initial submission) computes NFoO\mathrm{NFoO} on all sub-pools of size 6-16 and shows that on sub-pools where effective rank selects poorly, the severity of its selection is disproportionately worse compared to other proxies. This is additional evidence that effective rank is unreliable in this setting.


W2.2: “Is it true that: the games where a proxy fails differ from one proxy to another, and the proxy that works best differs from one game to another?”

Yes, it is true. On a given game, the median difference between best and worst proxies is an NFoO\mathrm{NFoO} of 0.49. Our initial prose focused only on aggregate level behavior; we have updated claims in the introduction, discussion, and Section 5.3 to reflect the fact that within games, failures are proxy-specific (see next comment for additional details).


W2.2/Q2: "'The choice of proxy matters little' is too absolute."

We thank the reviewer for carefully reviewing our claims and evidence! Our initial submission reported per-game results but drew its claims at the aggregate level. We now analyze per-game failure modes in more detail and state our claims separately at each level, to make the difference between them explicit. We’ve updated claims in the Abstract, Introduction, Results, and Discussion sections to reflect the following sentiment: among the proxies we consider, all select similarly well (and well above random) in aggregate, whereas within games, failures are proxy-specific.

In support of these claims, we also have extended our analysis in Appendix H: we recompute NFoO\mathrm{NFoO} on all sub-pools of size 6 to 16 and report (1) the difference in NFoO\mathrm{NFoO} between the sub-pool and the full pool of size 16, (2) the fraction of sub-pools where this difference is less than -0.1, and (3) the gap between best and worst proxies on each sub-pool. We find the following:

  • Selection quality is not a pool artifact. Every proxy’s NFoO\mathrm{NFoO} is robust to moderate changes in the pool, under both learners and within individual games. N-step return is the only proxy robust in every setting.

  • The comparison between proxies is not a pool artifact. Across methods, proxies remain similar to one another (particularly when 2 selections are allowed, i.e. NFoO2\mathrm{NFoO}_2). Within a game, they still remain substantially different.

Our recommendations on which proxies to use thus now reflect the setting in which selection is being performed: in aggregate (e.g. when choosing a pretraining method), using any of the probing-based proxies considered will suffice, but per-game, n-step return is the safer choice as it is the only proxy robust across all settings we consider.

AuthorsOpenReview ↗

Rebuttal [Part 3 of 3]

W3.1/Q3: “(a) Suppose I want to study whether intrinsic dimension is a good proxy of representation quality. Beyond “report both ρ\rho and NFoO\mathrm{NFoO},” how does this work help me along the way?”

Our study suggests the following:

  1. Report ρ\rho and NFoO\mathrm{NFoO} on a benchmark pool
  2. Perturb the pool. Recompute NFoO\mathrm{NFoO} on sub-pools, as in Appendix H, to measure how often and how badly the proxy’s selection fails when the candidates change.
  3. Investigate both in aggregate over tasks and per-task: aggregation can hide within task failures.
  4. Investigate what drives disagreements with downstream performance. In the case of effective rank, we could find that the two ST-DIM variants account for most of the gap.
  5. Compare against existing proxies. The seven proxies in this work provide a comparator for how well the new proxy performs under both ρ\rho and NFoO\mathrm{NFoO}. Computing its shared error with them (Section 5.5) further shows whether it captures similar or different structure from the existing proxies.

Evaluating a new proxy can be expensive mainly because of finetuning. Our code repository is already publicly available, but we are also exploring releasing all pretrained encoder weights. With these, a new proxy such as intrinsic dimension could be evaluated against our full set of finetuning runs, including ρ\rho, NFoO\mathrm{NFoO}, and the pool-perturbation analysis, without the burden of finetuning. We hope this helps address the reviewer’s concerns about the practical value of this work.


W3.2/Q3: “(b) Suppose I want to choose a good pretrained representation for FlashSAC on HumanoidBench, or for PQN on Atari. What selection procedure does this work suggest?”

  1. Use a proxy rather than none. In every setting we study, every proxy selects far above chance. Determine which proxies are computable from the data available.
  2. Use n-step return as a safer choice when reward labels are available. It has the highest per-game median NFoO\mathrm{NFoO}, never falls below random on any game, and is the only proxy robust to pool perturbation in every setting (Section 5.3, Appendix H).
  3. Shortlist the candidates with proxy. Finetuning the top two or three candidates recovers most of the achievable performance: every proxy reaches NFoO2\mathrm{NFoO}_2 at least 0.80 across methods. Because proxies fail on different games, computing several proxies is also informative. Where they agree on the top candidate, it can be trusted; where they disagree, finetune each proxy’s top pick.

[Andriopoulos et al., 2026] Geometric Analysis of Neural Regression Collapse via Intrinsic Dimension

[Todorov & Sabatelli, 2026] Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning


We hope that our responses address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you. Thank you again for the review!

Reviewer_DmZ5OpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the evaluation of representation-quality proxies in reinforcement learning. Existing work commonly evaluates such proxies using rank correlation with downstream RL performance. The authors argue that rank correlation does not necessarily reflect the practical objective of model selection, where identifying high-performing representations matters more than accurately ranking all candidates.

To address this mismatch, the paper proposes Normalized Fraction-of-Oracle (NFoO), a selection-oriented metric that measures the downstream performance of proxy-selected representations relative to the best and worst candidates in a given pool. The authors conduct a large-scale empirical study of seven representation proxies across 15 Atari games, multiple pretraining methods, and two downstream RL algorithms, Rainbow and PPO.

The main finding is that different proxies can exhibit substantially different rank correlations while achieving similarly strong selection performance. The paper further demonstrates the use of proxies for selecting representations within individual games and for early stopping during pretraining.

Critical strengths
  1. A well-motivated distinction between ranking and selection. The paper identifies a meaningful mismatch between the conventional evaluation of representation proxies and their intended practical use. Rank correlation measures the quality of the entire ordering, whereas model selection depends primarily on identifying high-performing candidates. The distinction is conceptually important, and the paper provides concrete examples showing that a proxy with relatively poor rank correlation can nevertheless select the best available representation.

  2. Interesting empirical findings supported by a substantial evaluation. The paper presents an extensive experimental study across different representation proxies, pretraining methods, and downstream RL learners. A particularly interesting finding is that proxy rankings can vary considerably across evaluation settings, while their model-selection performance remains relatively stable. The experiments also suggest that a wide range of proxy signals can be useful for selecting representations, despite capturing different properties. These observations provide useful empirical insights into how representation proxies should be evaluated and interpreted.

Critical weaknesses
  1. The central empirical finding requires deeper analysis and stronger evidence of robustness.

One of the paper's main conclusions is that proxies with substantially different rank correlations can nevertheless achieve similarly strong selection performance. Although this observation is interesting, it is unclear to what extent it reflects a general property of representation proxies rather than the particular candidate pools considered in the experiments. For example, Section 5.1 shows that effective rank achieves perfect top-1 selection despite having relatively poor rank correlation, but the authors also acknowledge that this result is partly fortunate and sensitive to the ranking of certain ST-DIM variants. This raises an important question about whether the observed selection robustness would persist under different candidate pool compositions, particularly when the top candidates are more difficult to distinguish. A more systematic analysis of the conditions under which ranking and selection disagree would substantially strengthen the scientific contribution.

  1. Limited understanding of why different proxies exhibit similar selection performance.

The paper evaluates seven proxies that capture different aspects of representations, including reward information, transition dynamics, temporal structure, and effective rank. An interesting observation is that their selection performance is often similar despite considerable differences in ranking quality. However, the paper provides limited explanation for why this occurs. It remains unclear whether these proxies capture shared characteristics of high-performing representations, whether the strongest candidates are relatively easy to identify, or whether the normalization and candidate pool structure contribute to the observed similarity. Without further investigation, the empirical findings remain somewhat descriptive rather than providing deeper insight into representation quality and model selection.

Questions for authors
  1. Sensitivity to NFoO normalization. Since NFoO depends on the best and worst candidates in the pool, how sensitive are the conclusions to the normalization scheme and candidate pool composition? Would alternative normalization methods lead to similar findings?

  2. Related work on RL representation learning. The related-work section could be expanded to cover methods that jointly optimize representation learning and RL objectives through auxiliary tasks, rather than focusing primarily on separate representation pretraining. Relevant examples include:

    • CURL: Contrastive Unsupervised Representations for Reinforcement Learning (Laskin et al., ICML 2020).
    • Return-Based Contrastive Representation Learning for Reinforcement Learning (Liu et al., ICLR 2021).
    • Masked Contrastive Representation Learning for Reinforcement Learning (Zhu et al., TPAMI 2023).
3 comments

We thank the reviewer for carefully reviewing our paper. We greatly appreciate the feedback! Please see below our responses to the comments.

We have also updated the paper by qualifying our original claims in more detail and adding additional experiments in the appendices based on the feedback (updated text in red).


W1 & Q1 (Robustness to pool composition and normalization)

We thank the reviewer for carefully reviewing our claims and evidence! Our initial submission reported per-game results but drew its claims at the aggregate level. We now analyze per-game failure modes in more detail and state our claims separately at each level, to make the difference between them explicit. We’ve updated claims in the Abstract, Introduction, Results, and Discussion sections to reflect the following sentiment: among the proxies we consider, all select similarly well (and well above random) in aggregate, whereas within games, failures are proxy-specific.

In support of these claims, we also have extended our analysis in Appendix H: we recompute NFoO\mathrm{NFoO} on all sub-pools of size 6 to 16 and report (1) the difference in NFoO\mathrm{NFoO} between the sub-pool and the full pool of size 16, (2) the fraction of sub-pools where this difference is less than -0.1, and (3) the gap between best and worst proxies on each sub-pool. We find the following:

  • Selection quality is not a pool artifact. Every proxy’s NFoO\mathrm{NFoO} is robust to moderate changes in the pool, under both learners and within individual games. N-step return is the only proxy robust in every setting.

  • The comparison between proxies is not a pool artifact. Across methods, proxies remain similar to one another (particularly when 2 selections are allowed, i.e. NFoO2\mathrm{NFoO}_2). Within a game, they still remain substantially different.

Our recommendations on which proxies to use thus now reflect the setting in which selection is being performed: in aggregate (e.g. when choosing a pretraining method), using any of the probing-based proxies considered will suffice, but per-game, n-step return is the safer choice as it is the only proxy robust across all settings we consider. italicized text


Q1 (Normalization)

We thank the reviewer for this question. We first considered the unnormalized fraction of oracle (the selected score divided by the best score). It is not well defined in Atari, because downstream scores (both raw and human-normalized) can be negative in some games (e.g. Pong), which makes the ratio change sign or exceed one. Min-max normalization avoids this and gives NFoO\mathrm{NFoO} a direct reading: 1 is the best representation in the pool, 0 the worst, and 0.5 midway between them. Because it is an affine transformation of the scores within each pool, it leaves the comparison between proxies, and with random selection, unchanged. Its role is therefore to make NFoO\mathrm{NFoO} readable and comparable across pools, not to change the conclusions.


W2 (Deeper understanding of why proxies select similarly)

We thank the reviewer for this insightful comment. Our results partially address two of the hypotheses raised. (i) Shared characteristics: the shared-error analysis in Section 5.5 shows that, outside two nested pairs (n-step return and reward; next action and inverse action), the proxies err on largely different representations, so their similar selection does not arise from capturing the same characteristics. (ii) Pool structure: the similarity persists when NFoO\mathrm{NFoO} is recomputed on every sub-pool of 6–16 methods (Appendix H.3).

We have added to the Discussion that understanding why proxies select similarly in aggregate, and why they fail on particular games, is important future work. We believe our main contributions stand independently of this explanation: showing that ρ\rho and NFoO\mathrm{NFoO} can disagree, and finding, across a large-scale evaluation, that proxies select similarly well in aggregate while failing on different games.


Q2 (Related work on joint representation learning)

We thank the reviewer for the references. CURL is already cited in Section 2 (we corrected our citation from the ArXiv version to the ICML citation) and we have updated the Related Works by adding Liu et al. (2021) and Zhu et al. (2023). Thank you again for the great suggestion!


[Laskin et al., 2020] CURL: Contrastive Unsupervised Representations for Reinforcement Learning

[Liu et al., 2021] Return-Based Contrastive Representation Learning for Reinforcement Learning

[Zhu et al., 2023] Masked Contrastive Representation Learning for Reinforcement Learning


We hope that our responses address your concerns and are helpful in improving your rating. If you have any other comments or feedback, please let us know! We are looking forward to hearing back from you. Thank you again for the review!

Reviewer_DmZ5OpenReview ↗

Response

Thank you for the response. It addresses all of my concerns, so I have raised my score to 4 (Accept).

We thank the reviewer for their careful review of this paper and recommendation for acceptance!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.