- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Environment parallelization in reinforcement learning is nowadays a standard technique for accelerating policy training, yet its impact on off-policy replay buffer dynamics remains poorly understood. This paper theoretically and empirically investigates the main hyperparameters governing massively parallel off-policy RL by characterizing the replay buffer as a temporal mixture of policies controlled by the Update-to-Cycle (UTC) ratio and decomposing gradient variance into two main components. Across multiple GPU-accelerated simulators and continuous control tasks, the authors evaluate how varying these hyperparameters affects cross-scale learning consistency.
Critical strengths
The paper structure is really easy to follow, i appreciate the clarity and linearity of the manuscript. I found out that the definition of the two source of variance in the gradient error, are very intuitive and novel from the theoretically perspective of the massive parallel learning. Separating the empirical gradient error into an exploration/collection sampling error in the buffer, scaling as and mini-batch subsampling error from the buffer, scaling as provides a clear theoretical explanation for understanding the relationship between and . Overall the paper presents a considerable amount of experiments, enough for supporting the theoretical claims.
Critical weaknesses
While I found the text clearly readable, i think that the figures need a restyling, since they are too small with confounding labels. One Weakness above all is a confounding point that emerges from the concept of "swiping " fixing the ratio of . Indeed every figure has that ratio fixed but with gradient colors evaluate the environment returns. This aspect is present in all the figures and need some further discussion. Inspecting the multiscale sweeps (Figures 2, 3, and 4), the authors did not vary by scaling buffer size (as recommended in their own hyperparameter tuning guideline in Section 6). Instead, they held the per-environment buffer ratio constant and varied strictly by changing the rollout length / update frequency (including fractional steps down to ).This creates a severe experimental confound: changing via simultaneously alters the UTD ratio () and the ratio of gradient updates to environment interactions. For example, comparing () to () means the latter performs more gradient updates per environment step. Consequently, performance differences attributed to the replay buffer window () are confounded with the effects of high-frequency updates and sample efficiency. This is underscored by the author's own observation that FastTD3 results are notably noisy and non-monotonic, suggesting that varying rollout lengths alter the learning dynamics independently of the buffer composition.
Questions for authors
- From the defition of it's not clear how the swipe of its values is applied, due to the fixed ratio of . Can you clarify that figures?
4 comments
Final score
Thank you for the detailed response, the improved figure legibility, and for running the additional -sweep vs. -sweep experiments to address my main concern.
I appreciate that the authors explicitly investigated the confound between varying rollout length () and replay buffer size (), revised the hyperparameter tuning guidelines in Section 6 accordingly, and added a discussion of UTD-induced overfitting to the Limitations section.
However, the new comparison confirms the core issue raised in my review: varying the UTC ratio via versus does not yield invariant learning dynamics,most notably for FastTD3 at low UTC ratios, where the -sweep degrades performance substantially. While the authors attribute this to finite-buffer gradient variance, the fact that changing simultaneously alters the UTD ratio (and consequently overfitting and sample efficiency dynamics) means that the main experimental sweeps in Figures 2--4 still conflate replay buffer temporal composition with update frequency effects.
Because this confounding factor is embedded across the primary empirical validation and is only partially disentangled rather than resolved across the main experimental suite, I still have reservations about the soundness of the conclusions drawn from those sweeps. I therefore i will maintain my current rating.
Thank you for taking the time to review our additional experiments and the revised manuscript.
We also thank you for raising the issue of confounding factors that arise from how the UTC ratio is varied. Since , changing the UTC ratio always requires changing the buffer size , the UTD ratio , or both.
varying the UTC ratio via versus does not yield invariant learning dynamics, most notably for FastTD3 at low UTC ratios, where the -sweep degrades performance substantially.
We agree. In our additional experiments, the -sweep and the -sweep sometimes diverged. This revealed that the choice of the variable used to control the UTC ratio, as well as its range, is important for reproducing the learning curves determined by the population buffer distribution. This point was not included in our initial manuscript, and we believe that adding it as Section 5.4 means that our experiments no longer rely solely on the -sweep.
the fact that changing simultaneously alters the UTD ratio (and consequently overfitting and sample efficiency dynamics) means that the main experimental sweeps in Figures 2--4 still conflate replay buffer temporal composition with update frequency effects.
The degradation of FastTD3 at low UTC ratios occurred in the -sweep, where the buffer size was reduced while the UTD ratio was kept fixed. This result therefore does not indicate performance degradation caused by changes in the UTD ratio, such as overfitting, nor does it indicate a problem with the design of our main experiments, which vary the UTC ratio through .
Moreover, the most important purpose of our main experiments in Fig. 2 to 4 is not to explain the performance differences when the UTC ratio is varied within each subplot. Rather, it is to test whether the learning curves approach a common trajectory when are increased proportionally while the UTC ratio is held fixed. A key contribution of this paper is that, based on Section 4.1, we showed the consistency of learning curves across scales for multiple UTC ratios under conditions where the time window of the policy history in the buffer is matched. Also, the main focus of Section 5.5 is transferability, namely whether a UTC ratio that is effective at one scale remains effective at larger scales. The central question throughout Sections 4 and 5 is not the optimal UTC ratio itself, but how changes in scale affect learning behavior under a fixed UTC ratio.
In light of the above, we would greatly appreciate it if you could reconsider the contributions of our paper from the perspective of learning behavior across scales at a fixed UTC ratio.
Thank you again for your careful review and thoughtful comments, which have helped us strengthen our paper.
We would like to add a brief clarification to our previous reply.
In our first reply, as well as in the revised guideline and Limitations, we referred to a "high UTD ratio" as a possible cause of overfitting. We would like to clarify the ranges of the UTD ratios in our experiments.
In Section 5.2 (Fig. 4), the UTD ratios range from to for . In Section 5.4 (Fig. 5), the UTD ratios when varying range from to . Smaller values of yield higher UTD ratios in our sweeps. We highlight here because both FastTD3 (Seo et al., 2025) and FlashSAC (Kim et al., 2026) use parallel environments by default for MuJoCo Playground tasks.
Such small UTD ratios are common in existing implementations for massively parallel simulation. The official FlashSAC implementation uses for both IsaacLab and MuJoCo Playground. The official FastTD3 implementation uses for G1 and H1 in IsaacLab and for T1 in MuJoCo Playground. Rybkin et al. (2025), an important prior study in this area, sweep UTD ratios from to on Isaac Gym.
Studies of overfitting often use a single environment and examine substantially higher UTD ratios. Rybkin et al. (2025) also report that "overfitting increases with UTD" in their DeepMind Control analysis, where the UTD ratios in their grid search range from to . In the broader discussion on high UTD ratios, even higher ratios such as are also considered (Chen et al., 2021).
Thus, although the UTD ratio varies within our sweeps, its absolute value remains small in the settings specified above. We recognize, however, that the UTD ratio alone does not determine the extent of overfitting, which may also depend on factors such as batch size. We also recognize the importance of the confounding introduced by changes in the UTD ratio within our sweeps. We hope these values provide useful context for assessing concerns about overfitting and other effects associated with the UTD ratio.
Thank you for your thoughtful review. We appreciate your positive assessment of the presentation and your description of our approach as “characterizing the replay buffer as a temporal mixture of policies” and “decomposing gradient variance into two main components”. We are particularly pleased that you found the decomposition “very intuitive and novel”. We address your questions below.
The choice of varying L or R
Thank you for this fundamental comment on our experimental design. As summarized in your review, when changing τ at a fixed E, each subplot of Figs. 2 to 4 fixes R and varies L (L-sweep). Fixing L and varying R produces the same change in τ (R-sweep). In Step 3 of the guideline in Section 6, we stated "Sweep τ more finely by varying R while keeping E, L, and the other training settings fixed", which recommends the R-sweep.
To investigate whether the choice of sweep affects learning performance, we conducted additional experiments comparing the L-sweep and R-sweep. Fig. A and B show the results. To obtain τ∈{27,29,211}, the L-sweep fixes R=1024E and varies L∈{23,21,2−1}, while the R-sweep fixes L=2−1 and varies R∈{64E,256E,1024E}. We use a batch size of 8,192.
Fig. A shows the training curves and Fig. B the final returns. In most cases, the R-sweep and the L-sweep give the same learning curve shapes and final returns. However, for FastTD3 at low UTC ratios, the R-sweep sometimes degrades performance severely. Eq. (5) in Section 4.2 explains this. Both sweeps yield the same expected gradient, but adjusting τ by reducing R enlarges the first term, which is inversely proportional to the effective buffer size, and likely increases the gradient variance. In Figs. 2 to 4, we intended to show the effect of the gradient variance by changing the scale (E,R), and the effect of the expected gradient by changing τ within each subplot. We therefore fixed R and varied L to avoid changing the explicit 1/R factor in the finite buffer term within each subplot.
However, we recognize that this argument justifies the L-sweep only in the sense that it does not undermine our analysis of the expected gradient and its variance within the range of E, R, and L studied in this paper. A high UTD ratio can cause problems such as overfitting, and we do not incorporate this effect into our analysis. We added this to the Limitations.
Based on these considerations, we conclude the following.
Revisions
We apologize that the figures contained small text that was hard to read. Based on your review, we revised the manuscript as follows (changes in red).