acceptodds
Under review as a conference paper at ICLR 2027

Efficient Online Estimation of Achievable Goal Distributions During Training for Goal-Conditioned Reinforcement Learning

Abstract

Goal-conditioned reinforcement learning (GCRL) trains a single policy to achieve diverse goals. Central to GCRL is the achievable goal distribution , the distribution of goals that the current policy can achieve. Accurate estimation of helps curriculum learning methods in GCRL sample behavioral goals (the goals used to collect training data) of intermediate difficulty, which substantially improves training efficiency. Estimating online, however, is challenging: the distribution evolves throughout training, and the estimator must track this moving target with data that are both fresh and sufficient. Existing methods meet only one of these requirements: replay-buffer-based estimation requires no extra environment interaction but is biased toward where past policies happened to visit, whereas policy evaluation faithfully reflects the current policy but demands a large number of fresh environment samples for every estimate. We empirically find that evolves smoothly during training, so historical policy evaluation data remain informative for estimating the current . Exploiting this finding, we propose PE-GCRL (GCRL with **P**eriodic Policy **E**valuation), which interleaves training with a small number of policy evaluations and estimates by reusing the accumulated historical evaluation data. Experiments on four mainstream GCRL tasks show that, across multiple curriculum algorithms, PE-GCRL reduces the estimation error of by roughly an order of magnitude, keeps behavioral goals at the frontier of achievability, and improves learning speed and final success rates with less than 5% training-time overhead.

Primary area:
reinforcement learning
Keywords:
Goal-Conditioned Reinforcement Learning, Achievable Goal Distribution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m3PFOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

Goal conditioned RL requires the estimation of the goal distribution which is difficult. Current methods rely on historical data or estimation with many new samples. The authors propose an alternative by observing that this goal distribution evolves smoothly. They thus exploit this to use historical data and couple it a smaller number of new evaluations. They evaluate the method on 4 tasks using 2 simulation engines.

Critical strengths

none

Critical weaknesses

Th extension is based on the observation that teh goal distribution evolves smoothly. This is then used in the previous GCRL technique (black in Alg 1), and then use the empirical observation to modify the basic algorithm (blue in Alg 1). It appears that the extension is empirically motivated and not a significant improvement to the previous state of art in understanding or intuition. I consider the contribution borderline.

Questions for authors

many of the figures, example Figs 7 and 8 were hard to read.

2 comments

We thank the reviewer for the feedback; below we first address the question on figure readability and then the concern about the contribution.

Q: Many of the figures, e.g., Figs. 7 and 8, were hard to read.

Thank you for pointing this out. We have revised the figures in the updated PDF. Figure numbers below follow the revision, in which the original Figs. 7 and 8 are now Figs. 8 and 9.

  • Larger layout. The NN ablation (now Fig. 8) spans the full width. The bb ablation (now Fig. 9) and the density-estimator ablation (now Fig. 10, previously a quarter-width inset) are placed side by side in one row. We also split the original Fig. 3: the overlay of behavioral goals on pagp_{ag} spans the full width (Fig. 3), and the scatter plots of behavioral goals over training (Fig. 4) now share a row with the KL and α\alpha curves (Fig. 5, previously a half-width inset).
  • Self-contained captions. Captions now state the color encoding, e.g., in Fig. 8 the orange points are behavioral goals and the blue shading is the ground-truth pagp_{ag} (darker is denser), and in Fig. 9 colors denote values of bb. Subplot titles name the quantity, e.g., "Episodes per evaluation NeN_e" and "Temperature τ\tau" in Fig. 7.
  • Takeaway sentences. The captions of Figs. 1–3 and 7–10 now end with one sentence stating the main observation, so each figure can be read without the surrounding text.

W: The extension is based on the observation that the goal distribution evolves smoothly ... the extension is empirically motivated and not a significant improvement to the previous state of art in understanding or intuition. I consider the contribution borderline.

Thank you for the summary. Our contribution is more than a smoothness observation plus a small change to Algorithm 1; we clarify it in three points.

1. We identify where the bottleneck of online pagp_{ag} estimation lies: the data source, not data freshness. The error from reusing older data (drift, curves in Fig. 1) stays below the error of RB-based estimation (dashed line in Fig. 1), and the RB-based error shows no downward trend, remaining about an order of magnitude above the evaluation-based error (Fig. 2). This challenges the implicit assumption of RB-based curricula (e.g., MEGA, OMEGA) that RB achieved goals approximate pagp_{ag}, and explains their failures: behavioral goals missing the GOIDs (Fig. 3) and a distorted learning progress α\alpha (Fig. 5).

2. A new analysis turns this observation into a quantitative condition. Let pag(d)p_{ag}^{(d)} denote the achievable goal distribution of the policy dd evaluation rounds ago, and let Dd=DKL(pag∣pag(d))D_d = D_{\mathrm{KL}}(p_{ag} \\| p_{ag}^{(d)}) be the drift of pagp_{ag} over this lag (cf. Fig. 1). In the recency-weighted fit, let wdw_d be the share of the total weight carried by data from dd rounds ago (∑dwd=1\sum_d w_d = 1). For the weighted KDE p~ag\tilde{p}_{ag}, the expected KL error is bounded by twice the weighted drift ∑dwdDd\sum_d w_d D_d plus a statistical term that vanishes as the effective sample size grows; with enough data, the error converges to a limit no larger than the weighted drift itself (see our response to Reviewer euWp, W1, for details). In contrast, an RB-based estimator converges to the distribution pRBp_{RB} of achieved goals in the RB, so its asymptotic error is at least DKL(pag∣pRB)D_{\mathrm{KL}}(p_{ag} \\| p_{RB}), which does not vanish. Hence PE-GCRL has the lower asymptotic bias whenever the weighted drift is smaller than this RB bias, which is exactly the comparison shown in Fig. 1. The full statement and proof will be added in a new appendix section.

3. The change is small by design, which makes it general. PE-GCRL replaces only the data source, the part our analysis identifies as the bottleneck. The same change lowers the estimation error under OMEGA, ODISCERN, and ORIG, adds under 5% training time (Table 1), and is insensitive to the density estimator (Fig. 10).

We will revise the contribution list in the Introduction to state point 1 explicitly and add the analysis in point 2 to the appendix.

Dear Reviewer m3PF,

As the discussion period is closing, could you let us know whether our response resolves your concern about the contribution? We would be glad to clarify anything further.

Reviewer_WpxoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper estimates the achievable goal distribution for GCRL curricula from occasional policy evaluations, reusing past evaluations with recency weights instead of fitting replay-buffer data. It reports much lower estimation error, better behavioral goals and higher success rates at little extra wall-clock cost.

Critical strengths

The diagnosis that the data source, rather than its freshness, is the bottleneck is convincing and cleanly isolated, and tracing estimation quality through goal selection to curriculum progress is useful. The method is simple and released with code.

Critical weaknesses
  1. The efficiency claim does not hold on a sample basis. Evaluation interactions are not counted in the learning curves even though they are substantial, and on the flight-control task they exceed the training interactions. The small wall-clock overhead also relies on running evaluations in parallel while training runs in a single environment.
  2. The error metric favors the method. The ground truth is built from the same kind of evaluation data with the same type of estimator, whereas the replay-based estimate targets a different distribution, and the order-of-magnitude claim is only shown on a single task.
  3. The recency weighting, the main new ingredient, is not shown to matter. The authors' own ablation favors almost uniform reuse while the default decays much faster, a zero-cost estimator from training rollouts is missing, and the curriculum's mixing coefficient changes along with the estimate, so the gains are not clearly attributed.
  4. The theoretical motivation is weak. The autoregressive argument is explicitly not a proof, stationarity holds only in part of the runs, and modeling detrended residuals does not say how stale samples should be weighted.
Questions for authors
  1. Could you plot success against total environment steps (training plus evaluation), especially on VVC, and report wall-clock time with the same parallelism for training and evaluation? Are the success-rate evaluations in Fig. 5 the same rollouts as EVALUATE, and are they counted for the baselines? Please also explain the VVC difference between Table 1 (42,239 s) and Table 6 (44,332 s).
  2. In App. D.1, does "KDE on the achieved goals" refer to the successful grid goals or to the final positions reached during evaluation? Do the ground truth and PE-GCRL share the kernel and bandwidth, and are goal dimensions normalized before fitting? An estimator-free metric, such as the Brier score of predicted success on held-out grid goals, and a per-task numerical table would make the comparison more convincing. Brier score: G. W. Brier. Verification of Forecasts Expressed in Terms of Probability.
  3. What is the unit of t? The values 4.481, 9.491, 19.496 and 99.499 equal -1/ln q for q = 0.8, 0.9, 0.95 and 0.99, which suggests a per-evaluation decay. Please report success rates for the τ sweep and for the limits τ → 0 and τ → ∞, and explain why τ = 9.491 is the default.
  4. How does an estimator built from the (behavioral goal, success) labels of ordinary training rollouts compare, at no extra interaction cost? Could you control for α, for example by tuning b so that OMEGA-RB follows OMEGA-PE's α trajectory, or by comparing MEGA-RB and MEGA-PE without α mixing? Do the main OMEGA runs use hindsight relabeling, given that App. E introduces HER as a separate variant built on GC-SAC (L949)?
  5. Which stationarity test and significance level were used in App. B, how was the trend estimated, and how was the AR order chosen? If the weighting cannot be derived more rigorously, I would suggest presenting it as a heuristic.
7 comments

We thank the reviewer for the thorough and constructive feedback. Below we respond to each question in order.

Q1 & W1

We provide both the total-step plots and parallelism-matched wall-clock breakdowns; we also clarify why training steps remain a natural primary metric and explain the VVC run-time discrepancy.

  1. plot success against total environment steps (training plus evaluation), especially on VVC.

    We replot Fig.5 with the x-axis counting all environment steps — training and evaluation combined: Reach(link), Push(link), Slide(link), VVC(link). On Reach and Push, PE-GCRL retains its advantage under this metric. On VVC, the success rate over total environment steps rises more slowly, but converges to a higher final level. This is because PE-GCRL can estimate pagp_{ag} more accurately, therefore samples goals that are more informative for policy updates, yielding better asymptotic performance.

    We also want to clarify why the original x-axis remains a natural primary metric. The steps counted in Fig.5 are those collected by the ROLLOUT function in Algorithm 1 — steps that are directly used to update the policy. Evaluation rollouts are diagnostic and are not directly used to update the policy. This matches the convention in standard RL benchmarking, where sample efficiency is measured against training interactions. To make this distinction explicit, we will relabel the x-axis from "Environment Steps" to "Training Env. Steps." and add the above figures into our manuscript.

  2. report wall-clock time with the same parallelism for training and evaluation

    We report eval rollout time (s) across parallelism levels on the Push and VVC tasks:

    Number of Parallel Eval Environments124816
    Push19201025 (1.87×)790 (2.43×)500 (3.84×)371 (5.18×)
    VVC1856311026 (1.68×)6440 (2.88×)3681 (5.04×)2361 (7.86×)

    Under matched parallelism (1 eval env), the per-component breakdown (s) is:

    Train RolloutOptimize PolicyEval RolloutEstimate DistributionTotal
    Push27723703719208741816
    VVC4366374031856320260543

    Even at 1 eval env, the policy evaluation overhead remains smaller than policy optimization. On Push, eval rollout accounts for only 4.59% of total training time (1920 / 41816). On VVC, the fraction is 30.66%. Although VVC incurs a larger relative overhead due to its longer episode length, our method improves the final success rate on VVC from below 60% to above 70%; we consider this overhead acceptable given the performance gain.

    To test whether the evaluation steps on VVC are well spent, we gave the baselines the same total interaction budget: we trained OMEGA-RB and GC-SAC for 3.5×1063.5 \times 10^6 environment steps (5 seeds each). This matches the ∼3.4×106\sim 3.4 \times 10^6 total steps of OMEGA-PE (1×1061 \times 10^6 training + 2.4×1062.4 \times 10^6 evaluation). The result is shown in this figure, where OMEGA-PE is plotted against its total environment steps (training + evaluation). Neither baseline benefits from the extra training. OMEGA-RB peaks at about 60% around 0.7×1060.7 \times 10^6 steps and then drifts down to about 48% at 3.5×1063.5 \times 10^6 steps. GC-SAC peaks at about 60% around 1.4×1061.4 \times 10^6 steps and then declines to about 30%. OMEGA-PE is slower at first, but it stays ahead of both baselines from about 2.3×1062.3 \times 10^6 total steps onward and ends at about 71%. At least on VVC, under the same total environment-step budget, spending a certain number of steps on policy evaluation yields a higher final success rate than spending them on additional training. The baselines' lower success rate is therefore not due to a smaller interaction budget: more training steps do not close the gap. On long-horizon tasks such as VVC, the evaluation overhead is a worthwhile use of the interaction budget.

  1. Are the success-rate evaluations in Fig. 5 the same rollouts as EVALUATE, and are they counted for the baselines?

    For PE-GCRL, the success-rate curves in Fig. 5 are computed from the same EVALUATE rollouts used by the algorithm to estimate p~ag\tilde{p}_{ag}. For baseline methods, we run an equivalent periodic evaluation purely for monitoring; these rollouts are not part of their training algorithms. In both cases, only steps from the ROLLOUT function (Algorithm 1) — those that enter the replay buffer and drive gradient updates — are counted on the x-axis. The monitoring evaluations are excluded from the x-axis for all methods, keeping the training-step comparison consistent across methods. Because the baselines' monitoring evaluations are not part of their algorithms, they are also excluded from the total-step plots in point 1: there, the baselines' x-axis counts only training steps, while PE-GCRL's x-axis adds the EVALUATE steps that its algorithm requires.

  2. Please also explain the VVC difference between Table 1 (42,239 s) and Table 6 (44,332 s)

    The two figures were obtained on different servers with slightly different CPU clock speeds. Table 1 and Table 6 report results from independent runs on these two machines, and the ~5% difference in total time is consistent with the hardware gap. We will re-run the Table 6 experiments on the same machine used for Table 1 and update Table 6 accordingly to ensure a consistent comparison.

Q2 & W2

We clarify the KDE construction details and address the estimator-neutrality concern with an independent fixed-grid evaluation.

  1. In App. D.1, does "KDE on the achieved goals" refer to the successful grid goals or to the final positions reached during evaluation?

    "KDE on the achieved goals" refers to the desired goals of successful episodes, not the final positions. Since pagp_{ag} is defined over the goal space G\mathcal{G}, upon a successful episode we record the desired goal gg that the policy was tasked with achieving, rather than the agent's final position, so that the KDE fits a distribution directly over G\mathcal{G}.

    For example, in the Push task the desired goal is to push the object to (0.2,0.2)(0.2, 0.2), and the agent actually pushes it to (0.18,0.22)(0.18, 0.22). With the achievement threshold δ=0.05\delta = 0.05, (0.18−0.2)2+(0.22−0.2)2=0.028<δ\sqrt{(0.18-0.2)^2 + (0.22-0.2)^2} = 0.028 < \delta, so the policy is deemed to achieve (0.2,0.2)(0.2, 0.2), and this desired goal is recorded as the achieved goal. The final position (0.18,0.22)(0.18, 0.22) is not used; using it would shift the KDE onto the continuous state space rather than the goal space.

  2. Do the ground truth and PE-GCRL share the kernel and bandwidth, and are goal dimensions normalized before fitting?

    Yes. All three estimates — the ground truth, OMEGA-RB's p~ag\tilde{p}_{ag}, and OMEGA-PE's p~ag\tilde{p}_{ag} — use the same KDE with a Gaussian kernel and bandwidth 0.2. Holding the estimator fixed is deliberate: it makes the data source the only factor that varies across the three, so any difference in DKL(pag∣p~ag)D_{KL}(p_{ag} \\| \tilde{p}_{ag}) reflects the data used to estimate pagp_{ag}, not a more favorably tuned estimator.

    Goal dimensions are normalized before fitting. We apply sklearn's StandardScaler to the goals before fitting any KDE.

  3. An estimator-free metric / The error metric favors the method

    We followed your suggestion and considered the Brier score, but find it less appropriate here: Brier score evaluates the calibration of a per-goal binary success predictor, whereas pagp_{ag} is a density over the goal space and the question is whether the full distribution is accurately recovered — a task for a distributional divergence rather than a pointwise predictor score.

    To obtain an estimator-neutral evaluation, we adopt a fixed discrete grid. On the Push task we select 676 equally-spaced anchor goals covering the goal space, run policy evaluation on each anchor, and record the empirical success indicator. This gives a ground-truth discrete distribution paggridp_{ag}^{\text{grid}}. We then evaluate the KDE estimates of OMEGA-PE and OMEGA-RB at the same 676 points, normalize each to a discrete distribution, and compute the KL divergence against paggridp_{ag}^{\text{grid}}. Because the anchor grid is fixed independently of any method's training data, the comparison is fair to both. The result is shown in this figure: OMEGA-PE's estimate remains significantly closer to the ground truth than OMEGA-RB's under this metric, consistent with the continuous-KDE results in the paper. We ran this experiment on Push to stay consistent with the ablation studies in the manuscript, all of which are conducted on Push (Section 4.5); we will extend it to all four tasks and report the results in the appendix.

    The reviewer notes that all three estimates share the same KDE estimator. As explained in point 2, this is a deliberate design choice, not a source of bias. The table below summarizes the setup:

    Estimating p~ag\tilde{p}_{ag}Data SourceEstimator
    OMEGA-RBReplay BufferKDE
    OMEGA-PEPolicy Evaluation with Recency-Based WeightingKDE
    Ground TruthPolicy EvaluationKDE

    The reviewer also notes that the replay-based estimate targets a different distribution. We agree, and this is what the comparison is meant to measure. Curriculum methods such as OMEGA and MEGA use the replay-buffer estimate as a stand-in for pagp_{ag} when selecting behavioral goals. The achieved goals in the buffer follow a different distribution from pagp_{ag}, because they mix states visited by past policies and reflect the behavioral goals the curriculum chose. Measuring DKL(pag∣p~ag)D_{\mathrm{KL}}(p_{ag} \\| \tilde{p}_{ag}) therefore quantifies how much bias the curriculum inherits from this stand-in. It does not penalize the replay-based estimate for targeting something other than what it is used for.

  4. the order-of-magnitude claim is only shown on a single task.

    The order-of-magnitude difference holds across all four tasks. Figs. 19, 20, 21, and 22 in Appendix E.2 show DKL(pag∣p~ag)D_{\mathrm{KL}}(p_{ag} \\| \tilde{p}_{ag}) for OMEGA-PE and OMEGA-RB on Reach, Push, Slide, and VVC, respectively. In each case, OMEGA-PE's estimation error is much lower than OMEGA-RB's throughout training, consistent with the Push result shown in the main paper.

Reviewer_euWpOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

In this manuscript, the authors points out the dilemma in existing approaches for goal‑conditioned reinforcement learning (GCRL): RB-based estimation requires suffer from bias while fresh policy evaluation requires large amount samples. The author claims pagp_{ag} evolves empirically and proposes PE-GCRL, which historical evaluation samples are reused for estimating pagp_{ag}. Experiments on several mainstream GCRL tasks demonstrate the superiority of the proposed method.

Critical strengths
  • The authors point out the critical issue in GCRL. It pinpoints that RB‑based bias originates from mismatched data source rather than insufficient density‑model capacity.

  • PE‑GCRL is straightforward, and it only swaps the data source for pagp_{ag} estimation. It is compatible with many other GCRL methods.

  • The manuscript is motivated from empirical studies. The experiments and empirical studies are comprehensive in this manuscript.

Critical weaknesses
  • Although the empirical studies are well studied, the manuscript lacks rigorous theoretical guarantees. The assumption are over relied on empirical observations.

  • It seems that the evaluation buffer grows to very large, and it will be costly for storage and density estimation. Is there any study on maximum buffer size setting or old‑sample eviction?

  • The proposed method miss some robustness tests against noise and policy oscillation. What if the success labels carry statistical noise or pagp_{ag} fluctuates back‑and‑forth, violating temporal‑smoothness assumption?

Questions for authors
  • It seems most experiments are based on low-dimension spaces. How is the performance of PE-GCRL when the dimension of goals are high?

  • The authors considers the smooth change of goal condition. It is also suggested to consider the jump in policy and changing environment, which could test the failure boundary of PE‑GCRL.

  • What about considering another baseline with τ\tau goes to 0, which discards all history and keeping only newest evaluation batch? It is suggested to consider this case in the ablation study.

6 comments

We sincerely thank the reviewer for the careful reading and the constructive suggestions. Below we respond to each question in order.

Q1: It seems most experiments are based on low-dimension spaces. How is the performance of PE-GCRL when the dimension of goals are high?

We introduce JointReach, a new Panda-Gym task where the goal is a 7-dimensional joint configuration target, to test PE-GCRL in a higher-dimensional setting. The source code is available at (link). The table below summarizes the goal space dimensions of JointReach and the four tasks in the manuscript:

TaskReachPushSlideVVCJointReach
Goal Space Dim.32237

Results (Figure). OMEGA-PE reaches ~100% success rate by ~300K steps and remains there for the rest of training. GC-SAC climbs slowly to ~70% at 1M steps without converging. OMEGA-RB stays near 0% throughout.

pagp_{ag} estimation error. We also measured DKL(pag∣p~ag)D_{\mathrm{KL}}(p_{ag} \\| \tilde{p}_{ag}) on JointReach, in the same two settings as Section 4.2: GC-SAC (Figure) and OMEGA (Figure). After the first ∼0.1×106\sim 0.1 \times 10^6 steps, when the evaluation buffer still holds little data and both PE curves spike, the PE-based estimate is more accurate than the RB-based one for the rest of training. In the GC-SAC setting, GC-SAC [PE] drops to about 2–3 and ends near 0.05, while GC-SAC [RB] declines slowly from above 10 and is still at about 1.6 at 1×1061 \times 10^6 steps. In the OMEGA setting, OMEGA-PE falls to near zero by ∼0.25×106\sim 0.25 \times 10^6 steps, while OMEGA-RB plateaus at about 6–7. This matches the success rates above: OMEGA-PE's error vanishes shortly before it reaches ∼100\sim 100\\% success, and OMEGA-RB's error stays high while its success rate stays near 00\\%.

Why OMEGA-PE remains effective at higher goal dimensionality. Two properties are key. First, evaluation data is collected directly in the desired goal space, with no need to project replay-buffer observations into it — a structural advantage that is independent of dimensionality. Second, the method is modular: for higher-dimensional goal spaces such as image-based goals, a goal encoder (e.g., CNN or VAE) can first map raw goals to a compact low-dimensional representation, with density estimation then performed in that representation space. As Section 4.5 analyzes, the estimator choice is flexible — KDE and GMM work well in low-dimensional representations, while neural estimators (NN, NF, FM) are the natural upgrade as representation dimensionality grows. Noise in the representation is not free either: as our response to W3 shows, it introduces a bias in p~ag\tilde{p}_{ag} that grows with the noise level, but at moderate noise OMEGA-PE degrades gracefully relative to the noise-free case rather than breaking down.

Why OMEGA-RB fails on JointReach. OMEGA-RB estimates pagp_{ag} by mapping replay buffer observations to goals. This requires that every observation can be mapped to a valid desired goal. In JointReach, however, not every observation in the replay buffer corresponds to a valid joint configuration target: an observation may fall outside the desired goal space entirely, or map to a configuration that is not a meaningful training goal. OMEGA-PE avoids this problem by construction — evaluation data is collected by querying the policy on explicitly sampled desired goals, so every entry in Be\mathcal{B}_e is guaranteed to be a valid desired goal.

We note that 7D represents a moderate increase in dimensionality; scaling to image-based or very high-dimensional goal spaces is an interesting direction for future work. Because the goal representation step and the choice of density estimator are both modular components, such extensions do not require changes to the core PE-GCRL algorithm.

Q2: The authors considers the smooth change of goal condition. It is also suggested to consider the jump in policy and changing environment, which could test the failure boundary of PE‑GCRL.

W3: The proposed method miss some robustness tests against noise and policy oscillation. What if the success labels carry statistical noise or pagp_{ag} fluctuates back‑and‑forth, violating temporal‑smoothness assumption?

We added two groups of experiments, one for each failure mode raised: (1) policy oscillation, where pagp_{ag} moves back and forth during training and violates the temporal-smoothness assumption, tested on a new PointMaze task; and (2) noisy evaluation data, where either the observed goals or the success labels are corrupted. In both settings, PE-GCRL keeps a lower pagp_{ag} estimation error than the RB-based estimator.

  1. Policy oscillation and pagp_{ag} back-and-forth fluctuation.

To test whether PE-GCRL remains reliable when pagp_{ag} fluctuates back-and-forth — directly violating the temporal-smoothness assumption — we introduce PointMaze, a maze navigation task with a fragmented desired goal space composed of many disconnected sub-regions (Figure). This structure makes local oscillations in pagp_{ag} visible and measurable.

Oscillation is real and substantial. We saved 100 checkpoints at equal intervals throughout GC-SAC training and evaluated each on all 1,125 discretized desired goals. Snapshots at 10%, 20%, and 30% training progress illustrate the phenomenon: at 110K steps the policy reaches 244/1,125 goals; by 210K steps this expands to 353/1,125 but several previously-reachable sub-regions are lost; by 310K steps coverage grows to 498/1,125 yet the pattern differs again. Such back-and-forth changes in which sub-regions are achievable are common throughout training, confirming that pagp_{ag} undergoes genuine oscillation rather than monotone expansion.

PE-GCRL tracks pagp_{ag} more accurately under oscillation. The DKL(pag∣p~ag)D_\text{KL}(p_{ag} \\| \tilde{p}_{ag}) curves for GC-SAC (Figure) and OMEGA (Figure) show that PE consistently achieves lower estimation error than the RB-based method throughout training. In the GC-SAC setting, GC-SAC [PE] converges to ~0.01 while GC-SAC [RB] stabilizes at ~0.07–0.09 and drifts upward in later training. In the OMEGA setting, OMEGA-PE remains below OMEGA-RB on average across the full run.

Why PE remains more accurate under oscillation. When pagp_{ag} swings back to a previous state, RB-based estimation suffers doubly: the replay buffer has accumulated a large volume of stale data without temporal decay, so the outdated samples contribute equally to the density estimate and cannot be suppressed. PE-GCRL's recency weighting (e−Δk/τe^{-\Delta k / \tau}) continuously down-weights older evaluation data, so the estimate responds faster to changes in pagp_{ag}. The oscillatory pattern visible in the OMEGA-PE KL curve in fact reflects this responsiveness — the estimator is tracking pagp_{ag} as it evolves rather than averaging over its history.

Adapting to severe oscillation. For tasks with particularly aggressive policy oscillation, one can reduce τ\tau or increase NeN_e to weight recent evaluation data more heavily, trading evaluation overhead for faster adaptation. As our response to Q3 shows, a smaller τ\tau shifts the estimator toward fresher data at the cost of a higher per-round evaluation budget — a knob that is not available to RB-based methods at all.

  1. What if the success labels carry statistical noise

Noise robustness. We tested two distinct noise models. In the first, independent Gaussian noise with standard deviation σ∈1.0δ,2.0δ,3.0δ\sigma \in \\{1.0\delta, 2.0\delta, 3.0\delta\\} is added to the observed goal during evaluation, where δ\delta is the goal-achieving threshold. In the second, each success label is independently flipped with probability p \in \\{1\\%, 5\\%, 10\\%, 30\\%\\}.

Goal-observation noise (Figure). Measured against noise-free OMEGA-PE, whose final DKL(pag∣p~ag)D_\text{KL}(p_{ag} \\| \tilde{p}_{ag}) is ∼0.13\sim 0.13, the final KL rises to ∼0.20\sim 0.20 at σ=1.0δ\sigma = 1.0\delta, ∼0.46\sim 0.46 at σ=2.0δ\sigma = 2.0\delta, and ∼0.80\sim 0.80 at σ=3.0δ\sigma = 3.0\delta — where the noise standard deviation is three times the success radius. The degradation grows steadily with the noise level, and OMEGA-PE stays below OMEGA-RB (∼2.15\sim 2.15) even at σ=3.0δ\sigma = 3.0\delta.

Success-label noise (Figure). Relative to the same noise-free baseline (∼0.13\sim 0.13), the final KL is ∼0.12\sim 0.12 at 1% flips, ∼0.15\sim 0.15 at 5%, ∼0.22\sim 0.22 at 10%, and ∼0.46\sim 0.46 at 30% — where nearly one in three evaluations carries the wrong outcome. Small flip rates change the estimate little, while the degradation becomes clear at 30%; OMEGA-PE remains below OMEGA-RB (∼2.15\sim 2.15) throughout.

The underlying reason is that p~ag\tilde{p}_{ag} is a population-level KDE estimate, not a single-sample query: the buffer accumulates hundreds to thousands of entries, and each individual noisy measurement contributes only a small weight to the aggregate. Per-sample noise does not simply average out, since flipped labels and perturbed goals introduce a systematic bias; however, at moderate noise levels this bias remains small relative to the true evaluation signal in the buffer, so OMEGA-PE degrades gracefully rather than collapsing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.