acceptodds
Under review as a conference paper at ICLR 2027

HE-FastSAC: Heterogeneous Training Objectives for Massively Parallel Off-Policy Reinforcement Learning

Abstract

Off-policy reinforcement learning has recently emerged as an alternative to on-policy methods on massively parallel simulators, where reuse of replay-buffer data improves sample efficiency. Current recipes in this regime optimize actor and critic under a single training objective, with hyperparameters tuned per task or per benchmark suite. Building on FastSAC, we present HE-FastSAC, which instead trains a population of policies under heterogeneous training objectives in a single run with minimal overhead. We partition the parallel environments into groups, each with its own target entropy and discount factor. A single actor and critic are shared across groups, and every group learns from the experience collected by all groups. We find that a policy learns faster and reaches a higher final score when trained inside this population than when trained alone under the same training objective. The gain comes both from the experience the population collects and from training a single network under multiple training objectives. Using one fixed hyperparameter set across 56 continuous-control tasks from four benchmark suites, with the same group deployed on every task, HE-FastSAC outperforms strong off-policy baselines and matches a per-task hyperparameter sweep of the same agent trained alone. Selecting a trained policy per task from the population yields a further gain with no additional training cost.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, massively parallel simulation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_BZv9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HE-FastSAC, an extension of FastSAC for massively parallel off-policy reinforcement learning. It partitions parallel environments into equal groups, each assigned a distinct target entropy and discount factor. A single actor and critic are conditioned on these two scalars and shared across groups. Cross-mixture relabeling exposes every group to transitions collected by all groups. Experiments cover 56 tasks from DMC/joystick, IsaacLab, HumanoidBench, and ManiSkill, with five seeds, comparing against FastSAC+RN+NR, FlashSAC, and FastTD3. A 12-task analysis separates heterogeneous collection from joint training.

Critical strengths
  1. The empirical validation is broad and carefully aggregated. Fifty-six tasks, five seeds, per-suite and per-task breakdowns, stratified bootstrap intervals, and an ablation ladder provide stronger evidence of generality than is typical in this area. This matters because the main claim is a population-level effect that should hold across training objectives and suites.
  2. The design is practical. Conditioning one network on two scalar descriptors avoids parameter growth with the number of groups, and cross-mixture relabeling keeps every update at the baseline batch size. The reported 2–11% overhead per iteration makes the method plausible for large simulators, and the HE-FlashSAC experiment suggests the mechanism is not tied to a single base learner.
Critical weaknesses
  1. The population, shared conditional network, shared replay, and cross-mixture relabeling are largely established motifs. The work combines them effectively but does not introduce a new principle.
  2. Relabeling transitions across discount factors and target entropies with uncorrected n-step targets lacks theoretical grounding. The “every group gains” claim is tested on only 12 tasks, and PQL is absent from the main comparison.
Questions for authors
  1. Why is PQL omitted from the main comparison, and can a matched environment/update-count comparison be provided?
  2. Can you quantify the bias of uncorrected relabeled targets, and would importance correction change the conclusions?
  3. How sensitive are the results to the number of groups, grid spacing, and environments per group?
  4. Can the “every group gains” analysis be extended to all 56 tasks?
  5. Does the shared actor/critic suffer interference across groups, and is there any gradient-conflict analysis?
3 comments

Thank you for the detailed review.

The population, shared conditional network, shared replay, and cross-mixture relabeling are largely established motifs. The work combines them effectively but does not introduce a new principle.

This is a fair summary. The paper shows how these components behave together in the massively parallel regime. Without the other groups' experience, the default group falls below FastSAC+RN+NR (Sec. 5.3), in contrast to NGU, whose members did slightly better on hard-exploration games when trained on their own data.

Relabeling transitions across discount factors and target entropies with uncorrected n-step targets lacks theoretical grounding.

Can you quantify the bias of uncorrected relabeled targets, and would importance correction change the conclusions?

We agree that uncorrected n-step returns are not technically correct for off-policy data, but they work well empirically and are standard practice in this regime (e.g., FastTD3 and PQL). To check that our conclusions do not depend on them, we reran HE-FastSAC and FastSAC+RN+NR with 1-step targets, which need no correction. Both methods lose some performance, but HE-FastSAC's gain stays the same (+0.074 vs. +0.072 mean final score on the 12 analysis tasks). We also measured the bias against the Retrace target (an importance-corrected n-step target; Munos et al., 2016): it is about 1% of the target's standard deviation, and the same for transitions from other groups as for a group's own. We have added both results to Appendix H of the updated paper.

The "every group gains" claim is tested on only 12 tasks

Can the "every group gains" analysis be extended to all 56 tasks?

Testing whether every group gains requires training each of the nine configurations alone on every task. For all 56 tasks at five seeds, this amounts to 2,520 runs, which is why we limited this analysis to the 12 representative tasks of Sec. 5.

PQL is absent from the main comparison.

Why is PQL omitted from the main comparison, and can a matched environment/update-count comparison be provided?

We did not include PQL in the main comparison, since it operates in a slightly different regime than the other baselines. Specifically, it is an asynchronous method that runs its actor and learners as separate processes, typically on multiple GPUs. FastTD3, which is included in our main comparison with the same environment and update counts as all other methods, acts as a good proxy though, since it can be seen as a synchronous version of PQL (i.e., a deterministic actor-critic with distributional critics, large batches, and per-environment noise scales). Its authors state this themselves: "we do not aim to claim novelty over PQL, but rather focus on developing a simple yet highly-optimized algorithm without asynchronous processes". Further, the paper includes an ablation of PQL's per-environment noise scale, which we found not to be beneficial in our case.

How sensitive are the results to the number of groups, grid spacing, and environments per group?

We have added a sensitivity analysis of the grid to the updated paper (Sec. 5.4). Since the total number of environments is fixed, the number of environments per group changes with the number of groups. With 4, 9 and 25 groups (down to 164 environments per group, and 5 on HumanoidBench), HE-FastSAC's improvement over FastSAC+RN+NR stays the same. Halving the grid spacing keeps about 85% of the improvement and doubling it about 60%, with HE-FastSAC clearly outperforming FastSAC+RN+NR for every grid.

Does the shared actor/critic suffer interference across groups, and is there any gradient-conflict analysis?

To check for gradient conflicts, we computed each group's gradient with respect to the shared actor and critic parameters and measured the cosine similarity between groups, corrected for minibatch noise (Appendix G in the updated paper). HE-FastSAC's groups point in partly different directions (0.38 for the critic and 0.50 for the actor, against 1 when splitting FastSAC+RN+NR's batch at random), mostly due to different discounts. However, no pair of groups conflicts on average, and a joint update helps each group almost as reliably as in the single-objective reference. In line with this, giving each group its own actor and critic does not improve over sharing them (Sec. 5.3).

Thank you again for your review. As the discussion period is coming to an end, if you have any further concerns, we’d be happy to address them.

Reviewer_BZv9OpenReview ↗

Keep positive score

Thanks for the response, which partially addresses my problems. As my rating is already positive, I will maintain it.

Reviewer_f55bOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes HE-FastSAC, a framework for massively parallel off-policy reinforcement learning that trains policies with heterogeneous entropy targets and discount factors using shared conditional actor-critic networks and a common replay buffer. Experiments across several continuous-control tasks demonstrate consistent improvements over recent off-policy RL baselines with modest computational overhead.

Critical strengths
  1. Consistent improvements across diverse benchmarks against recent and relevant baselines, including FastTD3 and FlashSAC, providing strong evidence of its effectiveness.

  2. The proposed method is intuitive, easy to implement, and readily extensible to existing off-policy RL frameworks without substantial architectural changes.

  3. The improvements come with minimal additional wall-clock overhead.

Critical weaknesses

The individual components, including conditional policies, heterogeneous training objectives, and shared replay, are closely related to existing approaches. The contribution primarily lies in their effective integration within massively parallel off-policy RL.

Questions for authors
  1. Does joint training remain beneficial when the entropy and discount configurations become more diverse or potentially conflicting?

  2. Additional experiments are not necessary, but I am curious whether the authors have considered comparing HE-FastSAC with more recent SAC-based methods, such as SimBaV2. Would HE-FastSAC remain competitive under matched environment implementations and training budgets?

2 comments

Thank you for your review and the positive assessment.

The individual components, including conditional policies, heterogeneous training objectives, and shared replay, are closely related to existing approaches. The contribution primarily lies in their effective integration within massively parallel off-policy RL.

We agree with this characterization.

Does joint training remain beneficial when the entropy and discount configurations become more diverse or potentially conflicting?

To make the groups' objectives more diverse, and thus potentially more conflicting, we have added a grid with twice the range of target entropies and discounts to the sensitivity analysis in the updated paper (Sec. 5.4). HE-FastSAC still clearly outperforms FastSAC+RN+NR there, keeping about 60% of its improvement. We also measured how well the groups' gradients in the shared networks align: with the wider grid they align less than with the paper's grid, but no pair of groups conflicts on average, and a joint update lowers each group's loss about as often as with the paper's grid (Appendix G).

Additional experiments are not necessary, but I am curious whether the authors have considered comparing HE-FastSAC with more recent SAC-based methods, such as SimBaV2. Would HE-FastSAC remain competitive under matched environment implementations and training budgets?

Matching SimBaV2's single-environment setting. Running HE-FastSAC with a single environment would need some consideration regarding which group to act with at any point, and how often to switch between groups. While we have not run any experiments, we do think it could be an interesting direction for future work.

Matching the massively parallel setting. Here, SimBaV2 would need adapting (e.g., batch size, update-to-data ratio, and replay buffer size). However, our main comparison includes FlashSAC under matched environment and update counts. FlashSAC combines many components of recent methods (e.g., the weight normalization and the distributional critic with reward scaling of SimBaV2, and the batch normalization of XQC and CrossQ), and its authors show that it outperforms SimBaV2 and XQC in the single-environment setting. In our main comparison, HE-FastSAC outperforms FlashSAC.

Thank you for the explanation and the supplementary experimental results, I will keep my current score.

Reviewer_QtjYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes HE-FastSAC. Under large-scale parallel simulation, it groups parallel environments, assigns each group a different target entropy and discount factor, and shares an actor and critic conditioned on these two values. Through random relabeling, each group is trained on experience from all groups. With little additional overhead, the method trains nine policies in a single run. On 56 continuous control tasks, using one fixed set of hyperparameters, it outperforms baselines such as FastSAC, FlashSAC, and FastTD3, and matches the performance of per-task tuning, thereby removing the need for per-task hyperparameter tuning.

Critical strengths
  1. The research problem is worthwhile. Training a separate policy for each task is costly, and the paper addresses this through a heterogeneous training approach.

  2. The experiments cover a broad range of tasks, 56 in total.

Critical weaknesses
  1. The paper is not very readable, and the writing needs further polishing.

  2. The novelty of the proposed method is limited. RN/NR comes from FlashSAC, cross-mixture comes from MEME, and the heterogeneous population comes from NGU/Agent57/CERL. The method design does not offer many standout contributions, and the method section is also fairly brief.

  3. All main results rely on the same 3×33\times3 grid. Would different grid ranges affect the experimental results?

Questions for authors

How are the different environments grouped, and does the grouping strategy affect model performance?

3 comments

Thank you for your review and questions.

The paper is not very readable, and the writing needs further polishing.

We have done additional rounds of writing revisions on the paper as a whole, which hopefully allow for easier reading and improved clarity. The main claims and results of the paper have not changed, and the changes mostly improve the flow of the text.

The novelty of the proposed method is limited. RN/NR comes from FlashSAC, cross-mixture comes from MEME, and the heterogeneous population comes from NGU/Agent57/CERL. The method design does not offer many standout contributions, and the method section is also fairly brief.

The individual components are indeed not new. RN and NR are taken from FlashSAC and are part of our baseline, FastSAC+RN+NR, so that the reported gains isolate the population (Sec. 3). Cross-mixture relabeling follows MEME, but our population is simpler, as each sample updates one group instead of every member, with no bandit and no reweighting. Beyond the benchmark results, we see the strength of the paper in its findings in this regime. Every group trains better inside the population than alone, and the gain depends on sharing experience across groups (Sec. 5.2, 5.3), whereas sharing was neutral or even harmful in NGU and in data-sharing ensembles (Lin et al., 2024). Separate networks on a shared buffer, as in CERL, recover only about half of the gain. The method section is short mainly because the method itself is simple. Sec. 3 of the updated paper now separates more clearly what we adopt from FastSAC and FlashSAC from what HE-FastSAC adds, and also states how environments are assigned to groups and how our relabeling differs from MEME's.

All main results rely on the same 3×33\times 3 grid. Would different grid ranges affect the experimental results?

We have added a sensitivity analysis regarding the grid choice to the paper (Sec. 5.4). In short, the improvement over FastSAC+RN+NR holds quite robustly across different choices of the grid (fewer and more groups, tighter spacing). The improvement is smallest when making the grid twice as wide (about 60% of the original), but even at that wide grid spacing, HE-FastSAC still clearly outperforms FastSAC+RN+NR.

How are the different environments grouped, and does the grouping strategy affect model performance?

We interleave the environments across groups, i.e., environment ii belongs to group i mod Ki \text{ mod } K. Each group is thus assigned an equal number of environments, and each environment keeps its group for the whole run. In most tasks the assignment does not matter, since all environments are identical up to randomness, so any assignment with equal group sizes is equivalent. It can matter when the environment index carries structure, e.g., on IsaacLab's rough terrains, where the curriculum assigns terrain types to contiguous blocks of environments; the interleaving gives every group the same terrain mix there. We have clarified this in the updated paper.

Thank you again for your review. As the discussion period is coming to an end, if you have any further concerns, we’d be happy to address them.

Thank you for the clarification and explanation. I will raise my original rating.

Comments on the paper

1 comment

We thank the reviewers for their feedback. We have incorporated the feedback and updated the paper. To summarize, the main changes are:

  • Sensitivity to the grid: We tested four additional grids with fewer and more groups and with halved and doubled ranges (Sec. 5.4). The default group outperforms FastSAC+RN+NR on every grid, and the gain is largely insensitive to the grid choice.
  • We measured the gradient alignment between the groups in the shared networks (Appendix G). No pair of groups has conflicting gradients, also with doubled ranges.
  • We added 1-step returns and measured the bias of the uncorrected n-step returns (Appendix H). The gain holds with 1-step returns, and the bias is about 1% of the target's standard deviation, no larger on other groups' transitions than on a group's own.
  • The method (Sec. 3): a clearer separation of what we adopt from what HE-FastSAC adds, the assignment of environments to groups, and the relation of our relabeling to MEME.
  • The relation of our findings to earlier populations and to data-sharing ensembles (Lin et al., 2024; Sec. 5.3 and 6).
  • Writing revisions throughout the paper, mainly for flow and clarity; the main claims and results are unchanged.

All reviewers note that the components of HE-FastSAC are not new, and we agree. We see the contribution in what their combination achieves in massively parallel off-policy RL: with one set of hyperparameters, a single run outperforms strong baselines on 56 tasks, and on the analysis tasks it matches tuning the target entropy and discount factor per task. Moreover, every policy of the population trains better than when it is trained alone, and this gain depends on sharing experience across groups, which some earlier populations found neutral or harmful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.