From Ranking to Selection: Evaluating Representation Proxies in RL

Pretraining representations can improve sample efficiency in reinforcement learning, but evaluating a representation normally requires expensive finetuning across many seeds and tasks. To avoid this cost, prior work scores representations with cheap proxies, such as linear probing against task-derived targets (e.g. reward) and direct measures of the representation itself (e.g. effective rank), and validates them by their rank correlation ($\rho$) with downstream performance. Such validation is incomplete, as $\rho$ scores the ordering of an entire pool of candidates, whereas the use case that matters in practice, model selection, depends only on the top choices. We therefore propose normalized fraction-of-oracle ($\mathrm{NFoO}$), which scores a proxy by the downstream performance of the representation it selects relative to the best available, and argue it should be reported alongside $\rho$. We also widen the pool of proxies under test: we evaluate a broad range of probing targets, spanning reward presence, multi-step return, behavioral cloning, episode structure, transition dynamics, and temporal structure. We assess seven proxies under both metrics in two model selection scenarios on the Atari benchmark: selecting among pretraining methods and among representations within a single game. We also evaluate their use for early stopping during pretraining. Across more than 20,000 finetuning runs on 15 games, we find an asymmetry: across proxies that capture different signals, proxy rankings shift with the candidate pool and downstream learner, yet their selections remain consistently strong. Representation proxies are therefore more robust as selectors than their rank correlations alone would suggest.