- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces HE-FastSAC, an extension of FastSAC for massively parallel off-policy reinforcement learning. It partitions parallel environments into equal groups, each assigned a distinct target entropy and discount factor. A single actor and critic are conditioned on these two scalars and shared across groups. Cross-mixture relabeling exposes every group to transitions collected by all groups. Experiments cover 56 tasks from DMC/joystick, IsaacLab, HumanoidBench, and ManiSkill, with five seeds, comparing against FastSAC+RN+NR, FlashSAC, and FastTD3. A 12-task analysis separates heterogeneous collection from joint training.
Critical strengths
- The empirical validation is broad and carefully aggregated. Fifty-six tasks, five seeds, per-suite and per-task breakdowns, stratified bootstrap intervals, and an ablation ladder provide stronger evidence of generality than is typical in this area. This matters because the main claim is a population-level effect that should hold across training objectives and suites.
- The design is practical. Conditioning one network on two scalar descriptors avoids parameter growth with the number of groups, and cross-mixture relabeling keeps every update at the baseline batch size. The reported 2–11% overhead per iteration makes the method plausible for large simulators, and the HE-FlashSAC experiment suggests the mechanism is not tied to a single base learner.
Critical weaknesses
- The population, shared conditional network, shared replay, and cross-mixture relabeling are largely established motifs. The work combines them effectively but does not introduce a new principle.
- Relabeling transitions across discount factors and target entropies with uncorrected n-step targets lacks theoretical grounding. The “every group gains” claim is tested on only 12 tasks, and PQL is absent from the main comparison.
Questions for authors
- Why is PQL omitted from the main comparison, and can a matched environment/update-count comparison be provided?
- Can you quantify the bias of uncorrected relabeled targets, and would importance correction change the conclusions?
- How sensitive are the results to the number of groups, grid spacing, and environments per group?
- Can the “every group gains” analysis be extended to all 56 tasks?
- Does the shared actor/critic suffer interference across groups, and is there any gradient-conflict analysis?
3 comments
Thank you again for your review. As the discussion period is coming to an end, if you have any further concerns, we’d be happy to address them.
Keep positive score
Thanks for the response, which partially addresses my problems. As my rating is already positive, I will maintain it.
Thank you for the detailed review.
This is a fair summary. The paper shows how these components behave together in the massively parallel regime. Without the other groups' experience, the default group falls below FastSAC+RN+NR (Sec. 5.3), in contrast to NGU, whose members did slightly better on hard-exploration games when trained on their own data.
We agree that uncorrected n-step returns are not technically correct for off-policy data, but they work well empirically and are standard practice in this regime (e.g., FastTD3 and PQL). To check that our conclusions do not depend on them, we reran HE-FastSAC and FastSAC+RN+NR with 1-step targets, which need no correction. Both methods lose some performance, but HE-FastSAC's gain stays the same (+0.074 vs. +0.072 mean final score on the 12 analysis tasks). We also measured the bias against the Retrace target (an importance-corrected n-step target; Munos et al., 2016): it is about 1% of the target's standard deviation, and the same for transitions from other groups as for a group's own. We have added both results to Appendix H of the updated paper.
Testing whether every group gains requires training each of the nine configurations alone on every task. For all 56 tasks at five seeds, this amounts to 2,520 runs, which is why we limited this analysis to the 12 representative tasks of Sec. 5.
We did not include PQL in the main comparison, since it operates in a slightly different regime than the other baselines. Specifically, it is an asynchronous method that runs its actor and learners as separate processes, typically on multiple GPUs. FastTD3, which is included in our main comparison with the same environment and update counts as all other methods, acts as a good proxy though, since it can be seen as a synchronous version of PQL (i.e., a deterministic actor-critic with distributional critics, large batches, and per-environment noise scales). Its authors state this themselves: "we do not aim to claim novelty over PQL, but rather focus on developing a simple yet highly-optimized algorithm without asynchronous processes". Further, the paper includes an ablation of PQL's per-environment noise scale, which we found not to be beneficial in our case.
We have added a sensitivity analysis of the grid to the updated paper (Sec. 5.4). Since the total number of environments is fixed, the number of environments per group changes with the number of groups. With 4, 9 and 25 groups (down to 164 environments per group, and 5 on HumanoidBench), HE-FastSAC's improvement over FastSAC+RN+NR stays the same. Halving the grid spacing keeps about 85% of the improvement and doubling it about 60%, with HE-FastSAC clearly outperforming FastSAC+RN+NR for every grid.
To check for gradient conflicts, we computed each group's gradient with respect to the shared actor and critic parameters and measured the cosine similarity between groups, corrected for minibatch noise (Appendix G in the updated paper). HE-FastSAC's groups point in partly different directions (0.38 for the critic and 0.50 for the actor, against 1 when splitting FastSAC+RN+NR's batch at random), mostly due to different discounts. However, no pair of groups conflicts on average, and a joint update helps each group almost as reliably as in the single-objective reference. In line with this, giving each group its own actor and critic does not improve over sharing them (Sec. 5.3).