Efficient Online Estimation of Achievable Goal Distributions During Training for Goal-Conditioned Reinforcement Learning

Goal-conditioned reinforcement learning (GCRL) trains a single policy to achieve diverse goals. Central to GCRL is the achievable goal distribution $p_{ag}$, the distribution of goals that the current policy can achieve. Accurate estimation of $p_{ag}$ helps curriculum learning methods in GCRL sample behavioral goals (the goals used to collect training data) of intermediate difficulty, which substantially improves training efficiency. Estimating $p_{ag}$ online, however, is challenging: the distribution evolves throughout training, and the estimator must track this moving target with data that are both fresh and sufficient. Existing methods meet only one of these requirements: replay-buffer-based estimation requires no extra environment interaction but is biased toward where past policies happened to visit, whereas policy evaluation faithfully reflects the current policy but demands a large number of fresh environment samples for every estimate. We empirically find that $p_{ag}$ evolves smoothly during training, so historical policy evaluation data remain informative for estimating the current $p_{ag}$. Exploiting this finding, we propose PE-GCRL (GCRL with **P**eriodic Policy **E**valuation), which interleaves training with a small number of policy evaluations and estimates $p_{ag}$ by reusing the accumulated historical evaluation data. Experiments on four mainstream GCRL tasks show that, across multiple curriculum algorithms, PE-GCRL reduces the estimation error of $p_{ag}$ by roughly an order of magnitude, keeps behavioral goals at the frontier of achievability, and improves learning speed and final success rates with less than 5% training-time overhead.