- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
Goal conditioned RL requires the estimation of the goal distribution which is difficult. Current methods rely on historical data or estimation with many new samples. The authors propose an alternative by observing that this goal distribution evolves smoothly. They thus exploit this to use historical data and couple it a smaller number of new evaluations. They evaluate the method on 4 tasks using 2 simulation engines.
Critical strengths
none
Critical weaknesses
Th extension is based on the observation that teh goal distribution evolves smoothly. This is then used in the previous GCRL technique (black in Alg 1), and then use the empirical observation to modify the basic algorithm (blue in Alg 1). It appears that the extension is empirically motivated and not a significant improvement to the previous state of art in understanding or intuition. I consider the contribution borderline.
Questions for authors
many of the figures, example Figs 7 and 8 were hard to read.
2 comments
Dear Reviewer m3PF,
As the discussion period is closing, could you let us know whether our response resolves your concern about the contribution? We would be glad to clarify anything further.
We thank the reviewer for the feedback; below we first address the question on figure readability and then the concern about the contribution.
Thank you for pointing this out. We have revised the figures in the updated PDF. Figure numbers below follow the revision, in which the original Figs. 7 and 8 are now Figs. 8 and 9.
Thank you for the summary. Our contribution is more than a smoothness observation plus a small change to Algorithm 1; we clarify it in three points.
1. We identify where the bottleneck of online pag estimation lies: the data source, not data freshness. The error from reusing older data (drift, curves in Fig. 1) stays below the error of RB-based estimation (dashed line in Fig. 1), and the RB-based error shows no downward trend, remaining about an order of magnitude above the evaluation-based error (Fig. 2). This challenges the implicit assumption of RB-based curricula (e.g., MEGA, OMEGA) that RB achieved goals approximate pag, and explains their failures: behavioral goals missing the GOIDs (Fig. 3) and a distorted learning progress α (Fig. 5).
2. A new analysis turns this observation into a quantitative condition. Let pag(d) denote the achievable goal distribution of the policy d evaluation rounds ago, and let Dd=DKL(pag∣pag(d)) be the drift of pag over this lag (cf. Fig. 1). In the recency-weighted fit, let wd be the share of the total weight carried by data from d rounds ago (∑dwd=1). For the weighted KDE p~ag, the expected KL error is bounded by twice the weighted drift ∑dwdDd plus a statistical term that vanishes as the effective sample size grows; with enough data, the error converges to a limit no larger than the weighted drift itself (see our response to Reviewer euWp, W1, for details). In contrast, an RB-based estimator converges to the distribution pRB of achieved goals in the RB, so its asymptotic error is at least DKL(pag∣pRB), which does not vanish. Hence PE-GCRL has the lower asymptotic bias whenever the weighted drift is smaller than this RB bias, which is exactly the comparison shown in Fig. 1. The full statement and proof will be added in a new appendix section.
3. The change is small by design, which makes it general. PE-GCRL replaces only the data source, the part our analysis identifies as the bottleneck. The same change lowers the estimation error under OMEGA, ODISCERN, and ORIG, adds under 5% training time (Table 1), and is insensitive to the density estimator (Fig. 10).
We will revise the contribution list in the Introduction to state point 1 explicitly and add the analysis in point 2 to the appendix.