acceptodds
Under review as a conference paper at ICLR 2027

What Governs Learning Dynamics in Massively Parallel Off-Policy RL?

Abstract

GPU-accelerated simulators enable data collection from thousands of environments in parallel for off-policy RL. This parallelism increases data throughput, but also changes the dynamics of the replay buffer distribution during the training. As the replay buffer has finite capacity, increasing data throughput causes its samples to be replaced more rapidly. In this study, we analyze the learning dynamics of reinforcement learning with massively parallel environments. Specifically, we treat the replay buffer as a temporal mixture of data distributions under changing policies and distinguish the true distribution shape from the sample density. We also identify two sources of gradient variance, one induced by replay buffer construction from finite rollout samples and the other by mini-batch sampling from the replay buffer. Our empirical results highlight the importance of increasing the replay buffer size and batch size as the number of parallel environments grows, and reveal regimes in which additional parallelism yields little benefit. These findings provide practical guidance for choosing replay buffer and mini-batch sampling hyperparameters in massively parallel off-policy RL.

Primary area:
reinforcement learning
Keywords:
off-policy reinforcement learning, experience replay, parallel reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bA72OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Environment parallelization in reinforcement learning is nowadays a standard technique for accelerating policy training, yet its impact on off-policy replay buffer dynamics remains poorly understood. This paper theoretically and empirically investigates the main hyperparameters governing massively parallel off-policy RL by characterizing the replay buffer as a temporal mixture of policies controlled by the Update-to-Cycle (UTC) ratio and decomposing gradient variance into two main components. Across multiple GPU-accelerated simulators and continuous control tasks, the authors evaluate how varying these hyperparameters affects cross-scale learning consistency.

Critical strengths

The paper structure is really easy to follow, i appreciate the clarity and linearity of the manuscript. I found out that the definition of the two source of variance in the gradient error, are very intuitive and novel from the theoretically perspective of the massive parallel learning. Separating the empirical gradient error into an exploration/collection sampling error in the buffer, scaling as 1κR\frac{1}{\kappa R} and mini-batch subsampling error from the buffer, scaling as 1KB\frac{1}{KB} provides a clear theoretical explanation for understanding the relationship between EE and BB. Overall the paper presents a considerable amount of experiments, enough for supporting the theoretical claims.

Critical weaknesses

While I found the text clearly readable, i think that the figures need a restyling, since they are too small with confounding labels. One Weakness above all is a confounding point that emerges from the concept of "swiping τ\tau" fixing the ratio of R/ER/E. Indeed every figure has that ratio fixed but with gradient colors evaluate the environment returns. This aspect is present in all the figures and need some further discussion. Inspecting the multiscale sweeps (Figures 2, 3, and 4), the authors did not vary τ\tau by scaling buffer size RR (as recommended in their own hyperparameter tuning guideline in Section 6). Instead, they held the per-environment buffer ratio R/ER/E constant and varied τ\tau strictly by changing the rollout length / update frequency LL (including fractional steps down to L=2−7L = 2^{-7}).This creates a severe experimental confound: changing τ\tau via LL simultaneously alters the UTD ratio (σ=1/EL\sigma = 1/EL) and the ratio of gradient updates to environment interactions. For example, comparing τ=128\tau = 128 (L=8L=8) to τ=8,192\tau = 8,192 (L=1/8L=1/8) means the latter performs 64×64\times more gradient updates per environment step. Consequently, performance differences attributed to the replay buffer window (τ\tau) are confounded with the effects of high-frequency updates and sample efficiency. This is underscored by the author's own observation that FastTD3 results are notably noisy and non-monotonic, suggesting that varying rollout lengths LL alter the learning dynamics independently of the buffer composition.

Questions for authors
  • From the defition of τ\tau it's not clear how the swipe of its values is applied, due to the fixed ratio of R/ER/E. Can you clarify that figures?
4 comments

Thank you for your thoughtful review. We appreciate your positive assessment of the presentation and your description of our approach as “characterizing the replay buffer as a temporal mixture of policies” and “decomposing gradient variance into two main components”. We are particularly pleased that you found the decomposition “very intuitive and novel”. We address your questions below.

The choice of varying LL or RR

Thank you for this fundamental comment on our experimental design. As summarized in your review, when changing τ\tau at a fixed EE, each subplot of Figs. 2 to 4 fixes RR and varies LL (LL-sweep). Fixing LL and varying RR produces the same change in τ\tau (RR-sweep). In Step 3 of the guideline in Section 6, we stated "Sweep τ\tau more finely by varying RR while keeping EE, LL, and the other training settings fixed", which recommends the RR-sweep.

To investigate whether the choice of sweep affects learning performance, we conducted additional experiments comparing the LL-sweep and RR-sweep. Fig. A and B show the results. To obtain τ∈{27,29,211}\tau \in \lbrace 2^7, 2^9, 2^{11} \rbrace, the L-sweep fixes R=1024ER=1024E and varies L∈{23,21,2−1}L \in \lbrace 2^3, 2^1, 2^{-1} \rbrace, while the R-sweep fixes L=2−1L=2^{-1} and varies R∈{64E,256E,1024E}R \in \lbrace 64E, 256E, 1024E \rbrace. We use a batch size of 8,1928{,}192.

Fig. A shows the training curves and Fig. B the final returns. In most cases, the RR-sweep and the LL-sweep give the same learning curve shapes and final returns. However, for FastTD3 at low UTC ratios, the RR-sweep sometimes degrades performance severely. Eq. (5) in Section 4.2 explains this. Both sweeps yield the same expected gradient, but adjusting τ\tau by reducing RR enlarges the first term, which is inversely proportional to the effective buffer size, and likely increases the gradient variance. In Figs. 2 to 4, we intended to show the effect of the gradient variance by changing the scale (E,R)(E, R), and the effect of the expected gradient by changing τ\tau within each subplot. We therefore fixed RR and varied LL to avoid changing the explicit 1/R1/R factor in the finite buffer term within each subplot.

However, we recognize that this argument justifies the LL-sweep only in the sense that it does not undermine our analysis of the expected gradient and its variance within the range of EE, RR, and LL studied in this paper. A high UTD ratio can cause problems such as overfitting, and we do not incorporate this effect into our analysis. We added this to the Limitations.

Based on these considerations, we conclude the following.

  • The LL-sweep is appropriate for preserving the finite-buffer gradient variance, and the RR-sweep is appropriate for preserving the UTD ratio.
  • Our additional experiments suggest that finite-buffer gradient variance can significantly affect learning performance, highlighting the importance of maintaining a sufficiently large replay buffer.
  • The guideline should sweep both RR and LL, avoiding a large finite-buffer variance or a high UTD ratio, rather than fixing either one. The previous step 3 of the guideline, which mentions only the RR-sweep, was incorrect, and we have revised it accordingly.

Revisions

We apologize that the figures contained small text that was hard to read. Based on your review, we revised the manuscript as follows (changes in red).

  • We enlarged the text in Figures 2 to 4, 6 to 9, and 14 (the first sentence of each caption is shown in red).
  • Section 4.2 now states that hyperparameter combinations giving the same UTC ratio can still yield different gradient variances.
  • Section 5.2 now defines which hyperparameters are swept in Figs. 2, 3, and 4.
  • We added a new Section 5.4 describing the R-sweep and the L-sweep (Q3). The former Section 5.4 (Q3) is now Section 5.5 (Q4).
  • We revised Step 3 of the guideline in Section 6.
  • We added a discussion of overfitting to the Limitations.
Reviewer_bA72OpenReview ↗

Final score

Thank you for the detailed response, the improved figure legibility, and for running the additional LL-sweep vs. RR-sweep experiments to address my main concern.

I appreciate that the authors explicitly investigated the confound between varying rollout length (LL) and replay buffer size (RR), revised the hyperparameter tuning guidelines in Section 6 accordingly, and added a discussion of UTD-induced overfitting to the Limitations section.

However, the new comparison confirms the core issue raised in my review: varying the UTC ratio via LL versus RR does not yield invariant learning dynamics,most notably for FastTD3 at low UTC ratios, where the RR-sweep degrades performance substantially. While the authors attribute this to finite-buffer gradient variance, the fact that changing LL simultaneously alters the UTD ratio (and consequently overfitting and sample efficiency dynamics) means that the main experimental sweeps in Figures 2--4 still conflate replay buffer temporal composition with update frequency effects.

Because this confounding factor is embedded across the primary empirical validation and is only partially disentangled rather than resolved across the main experimental suite, I still have reservations about the soundness of the conclusions drawn from those sweeps. I therefore i will maintain my current rating.

Thank you for taking the time to review our additional experiments and the revised manuscript.

We also thank you for raising the issue of confounding factors that arise from how the UTC ratio τ\tau is varied. Since τ=Rσ\tau = R\sigma, changing the UTC ratio always requires changing the buffer size RR, the UTD ratio σ\sigma, or both.

varying the UTC ratio via LL versus RR does not yield invariant learning dynamics, most notably for FastTD3 at low UTC ratios, where the RR-sweep degrades performance substantially.

We agree. In our additional experiments, the LL-sweep and the RR-sweep sometimes diverged. This revealed that the choice of the variable used to control the UTC ratio, as well as its range, is important for reproducing the learning curves determined by the population buffer distribution. This point was not included in our initial manuscript, and we believe that adding it as Section 5.4 means that our experiments no longer rely solely on the LL-sweep.

the fact that changing LL simultaneously alters the UTD ratio (and consequently overfitting and sample efficiency dynamics) means that the main experimental sweeps in Figures 2--4 still conflate replay buffer temporal composition with update frequency effects.

The degradation of FastTD3 at low UTC ratios occurred in the RR-sweep, where the buffer size was reduced while the UTD ratio was kept fixed. This result therefore does not indicate performance degradation caused by changes in the UTD ratio, such as overfitting, nor does it indicate a problem with the design of our main experiments, which vary the UTC ratio through LL.

Moreover, the most important purpose of our main experiments in Fig. 2 to 4 is not to explain the performance differences when the UTC ratio is varied within each subplot. Rather, it is to test whether the learning curves approach a common trajectory when (E,R)(E, R) are increased proportionally while the UTC ratio is held fixed. A key contribution of this paper is that, based on Section 4.1, we showed the consistency of learning curves across scales for multiple UTC ratios under conditions where the time window of the policy history in the buffer is matched. Also, the main focus of Section 5.5 is transferability, namely whether a UTC ratio that is effective at one scale remains effective at larger scales. The central question throughout Sections 4 and 5 is not the optimal UTC ratio itself, but how changes in scale affect learning behavior under a fixed UTC ratio.

In light of the above, we would greatly appreciate it if you could reconsider the contributions of our paper from the perspective of learning behavior across scales at a fixed UTC ratio.

Thank you again for your careful review and thoughtful comments, which have helped us strengthen our paper.

We would like to add a brief clarification to our previous reply.

In our first reply, as well as in the revised guideline and Limitations, we referred to a "high UTD ratio" as a possible cause of overfitting. We would like to clarify the ranges of the UTD ratios in our experiments.

In Section 5.2 (Fig. 4), the UTD ratios range from 2−192^{-19} to 2−72^{-7} for E≥1,024E\ge1{,}024. In Section 5.4 (Fig. 5), the UTD ratios when varying LL range from 2−172^{-17} to 2−92^{-9}. Smaller values of EE yield higher UTD ratios in our sweeps. We highlight E≥1,024E\ge1{,}024 here because both FastTD3 (Seo et al., 2025) and FlashSAC (Kim et al., 2026) use 1,0241{,}024 parallel environments by default for MuJoCo Playground tasks.

Such small UTD ratios are common in existing implementations for massively parallel simulation. The official FlashSAC implementation uses 2−92^{-9} for both IsaacLab and MuJoCo Playground. The official FastTD3 implementation uses 2−102^{-10} for G1 and H1 in IsaacLab and 2−92^{-9} for T1 in MuJoCo Playground. Rybkin et al. (2025), an important prior study in this area, sweep UTD ratios from 2−162^{-16} to 2−102^{-10} on Isaac Gym.

Studies of overfitting often use a single environment and examine substantially higher UTD ratios. Rybkin et al. (2025) also report that "overfitting increases with UTD" in their DeepMind Control analysis, where the UTD ratios in their grid search range from 202^0 to 232^3. In the broader discussion on high UTD ratios, even higher ratios such as 2020 are also considered (Chen et al., 2021).

Thus, although the UTD ratio varies within our sweeps, its absolute value remains small in the settings specified above. We recognize, however, that the UTD ratio alone does not determine the extent of overfitting, which may also depend on factors such as batch size. We also recognize the importance of the confounding introduced by changes in the UTD ratio within our sweeps. We hope these values provide useful context for assessing concerns about overfitting and other effects associated with the UTD ratio.

Reviewer_H9tFOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates how various hyperparameters, including the number of parallel environments, the rollout length, replay buffer size, and batch size, affect RL performance in massively parallel simulation environments. The main contribution of this work is its detailed analysis of how changes in these hyperparameters influence the learning dynamics of RL, ultimately providing practical guidelines for tuning these hyperparameters in massively parallel RL. In addition, I believe the experimental evaluation is quite comprehensive. With a total computational cost of approximately 7,000 GPU hours, the authors have conducted extensive experiments, and the presented results are sufficiently comprehensive.

Critical strengths

1.The paper investigates the impact of hyperparameter variations on RL training performance using two mainstream off-policy RL algorithms, SAC and TD3. The selection of these representative algorithms makes the experimental validation rigorous.

2.The experiments are conducted across three GPU-accelerated simulation platforms: IsaacGym, IsaacLab, and MuJoCo Playground. The extensive training and evaluations demonstrate the rigor of the experimental study. Through a large number of experiments, the paper provides concrete and practical guidelines for replay buffer sizing and hyperparameter selection in massively parallel off-policy RL.

Critical weaknesses

Fig. 2, 3, and 4 all use the number of critic update steps as the x-axis. Taking Fig. 4 as an example, increasing the number of parallel environments may also change the environment interaction cost, memory requirements, and wall-clock training time. Including measurements of the training time and memory consumption required to achieve comparable return levels would further enhance the practical value of this study.

Questions for authors

1.How do GPU memory consumption and wall-clock training time change as the number of parallel environments increases? Does increasing environment parallelism also increase the wall-clock time required for a single critic update? Providing a comparison of per-update time and actual training efficiency across different configurations would further enhance the practical value of this study.

2.Both the gradient variance analysis in Fig. 6 and the batch size experiments in Fig. 7 are conducted exclusively on the G1 task. Does the gradient variance analysis exhibit a similar trend on other tasks? Specifically, does the estimated replay buffer gradient variance also generally decrease as the number of parallel environments and the replay buffer size increase?

4 comments
AuthorsOpenReview ↗

Response to Reviewer H9tF (1/2)

Thank you for your thoughtful review. We appreciate your recognition of our focus on “how changes in these hyperparameters influence the learning dynamics of RL” and on “providing practical guidelines for tuning these hyperparameters.” We are also pleased that you recognized “the extensive training and evaluations.” We address your questions below.

Q1. Reporting wall-clock time and GPU memory

Thank you for this important and constructive comment. We originally used critic updates as the x-axis because the "Consistent learning dynamics" claim in Section 4.1 concerns consistency measured per model update. On the other hand, as you pointed out, increasing the computational cost (the number of parallel environments and the batch size) substantially changes wall-clock time and GPU memory usage. We fully agree that reporting these is important for practical, budget-aware hyperparameter selection.

How do GPU memory consumption and wall-clock training time change as the number of parallel environments increases?

We show our wall-clock time analysis in the two figures below.

(2 seeds on an A100 80GB. At the number of parallel environments E=65,536E = 65{,}536, the buffer was stored on the CPU due to memory limits, so the value is shown in parentheses in Fig. B.)

Fig. A shows training curves against wall-clock time for varying parallelism and UTC ratios, with the batch size fixed at B=8,192B=8{,}192. Fig. B shows, for each algorithm and task, the time to reach 80% of the best return at each level of parallelism, broken down into model updates and env steps.

We observe two trends in wall-clock time from Fig. A.

  1. Wall-clock time increases with the parallelism EE and the replay buffer size RR.
  2. Wall-clock time increases as the UTC ratio τ\tau decreases. This is most visible for FastTD3 on MJP T1.

Both are natural trends. A larger EE incurs more overhead. In addition, within each subplot we vary τ\tau under a fixed RR, so a smaller τ\tau requires more data collection (see our discussion with Reviewer bA72 on the validity of this setting).

Including measurements of the training time and memory consumption required to achieve comparable return levels

For the time to reach a certain return level, the dashed lines in Fig. A mark 80% of the best return across all parallelism settings for each algorithm and task. Fig. B shows the fastest time to reach this return, broken down into model updates and env steps. The shortest time (bold in Fig. B) is obtained at a moderate parallelism. We attribute this to the combination of two trends.

  1. At small parallelism, the fastest run sometimes uses a small UTC ratio, which requires more env-step time (notably FastTD3 on MJP T1 and FlashSAC on Isaac G1).
  2. At large parallelism, the same UTC ratio becomes optimal, and a larger parallelism then requires more env-step time.

For the GPU memory, Fig. C reports the throughput and GPU memory of the simulator alone under random actions, measured on IsaacLab (Isaac-Velocity-Flat-G1-v0) and MuJoCo Playground (T1JoystickFlatTerrain) with an A100 80GB and a V100 16GB.

For E>4,096E > 4{,}096, simulator memory grows approximately linearly with EE, while throughput gains diminish. During training, the simulator and replay buffer dominate memory usage. Buffer memory scales linearly with capacity, reaching 92 GiB for a 64M buffer on MJP T1 Flat.

Does increasing environment parallelism also increase the wall-clock time required for a single critic update?

Not necessarily. In off-policy RL, the cost of a model update is decoupled from the number of env steps. Our model update is compiled in advance as a CUDA Graph, so its execution time barely changes under a fixed batch size. Fig. B shows that the model-update time needed to reach the target return does not increase with parallelism.

AuthorsOpenReview ↗

Response to Reviewer H9tF (2/2)

Q2. Do the gradient variance and noise scale analysis and the batch size sweep show similar trends on other tasks?

Yes. Fig. D and E show the results on Isaac H1 and MJP T1. Results are averaged over 3 seeds, and shaded areas show the standard deviation.

In Fig. D, the buffer component of both the variance and the noise scale decreases as EE and RR increase, consistent with Eq. (5) in Section 4.2. However, the decrease can be small when EE and RR are small, and training sometimes fails in such cases. The mini-batch component tends to increase with EE. These trends are similar to those on G1.

In Fig. E, larger batch sizes yield better performance per critic update, except when EE is small. Overall, the results are similar to those on G1.

Revisions

Based on your review, we revised the manuscript as follows (changes shown in red).

  • We added a “Wall-Clock Time Analysis” section to the Appendix, describing the experimental setup and results.
  • We added the gradient variance and batch size results on the H1 and T1 tasks to Appendix B.

Thank you for your response and explanation. After going through all the responses and comments from other reviewers, I will keep my positive score for now.

Thank you for taking the time to review our responses and the broader discussion. We appreciate your constructive feedback, which helped us improve the practical guidance and strengthen the experimental evidence in the revised manuscript.

Reviewer_wKnLOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
1
Read review
Paper summary

This paper investigates the learning dynamics of highly parallelized off-policy reinforcement with a distinct focus on the data distributions contained in the replay buffer. The paper sets forth on providing a parametric study, akin to many scaling law analyses common across the ML community, of the effects of parallelization when training with off-policy RL algorithms. Summary results confirm the best practices of current top approaches when faced with highly parallelized off-policy settings.

Critical strengths

This paper sets out an ambitious parametric study to investigate the mechanisms of highly parallelized off-policy RL. I was very excited to see this paper because I believe that we need more rigorous analyses of this spirit to develop deeper understanding of where specific algorithmic approaches work best which can guide more informed algorithmic research in the future.

I found the conceptual framing of the replay buffer's temporal distribution to be very compelling and a nice foundation for the analysis executed in the paper. I wish that more was done to elaborate on the definitions (e.g. framing the analysis to probe specifically at the buffer distribution continuum as outlined in Figure 1(b)).

Critical weaknesses

This paper was extremely hard to follow as there was no overarching structure or clearly justification/motivation for the analyses executed in Sections 4 and 5. The "Preliminary and Notation" section was unnecessary because the definitions provided in the section are repeated (and augmented) several times in following sections. This, or they were not used again (e.g. replay ratio). The paper's structure was significantly lacking, there is little to no cohesion between concepts in Section 4 where I would imagine the authors would want to convey specific axes their empirical analysis would vary. If I were to be generous, the concepts and ideas represented in Equation 6 do this but they are not specified or clearly laid out.

The claim the authors make at the end of this paper providing a systematic study into the effects of high rates of parallelization. However, I do not feel that the experimental analysis appropriately introduce the axes of investigation nor are they rigorously carried out. Yes, there are sweeps that are correct experimentally but I felt that all of Section 5 was poorly structured and I had no real clear understanding of why the specific choices of experiments were made and how the research questions laid out were motivated or justified.

Detailed and rigorous analyses should be celebrated and acknowledged as major parts of our scientific study. However, I do not feel that this paper was at all close to sufficient for publication. It was unstructured, experimental analyses are poorly motivated, the takeaways and discussion is superficial, and there is a lot of unnecessary and unused theoretical analysis.

Questions for authors

I don't have any real questions for the authors. I do like the direction of this paper but found the execution to be significantly lacking. I would recommend a thorough revision and re-submission at a later time. I am usually a major proponent of giving authors the benefit of the doubt but the changes needed to have me adjust my perspective of this paper are significant enough that I do not feel that it would be possible for the authors to do so during the review period. As such, I do not foresee me raising my score above "weak reject". For now, this paper is very clearly a "reject" in my estimation.

4 comments

Thank you for your thoughtful review. We are glad that you described our paper as an “ambitious parametric study” and found the conceptual framing of “the replay buffer's temporal distribution to be very compelling and a nice foundation”.

We aimed to structure the analysis in Section 4 and the experiments in Section 5 based on a consistent motivation. In this rebuttal, we hope to clarify how the subsections of Section 4 connect and how each experiment in Section 5 corresponds to Section 4. 

The paper's structure was significantly lacking, there is little to no cohesion between concepts in Section 4

Thank you for this comment. Let us clarify the main point of Section 4. From the perspective of the replay buffer distribution, the two hyperparameters, the UTD ratio and the replay buffer size, determine the shape of the distribution (Section 4.1) and its sample density (Section 4.2). From the perspective of the gradient, Section 4.1 discusses its expectation and Section 4.2 its variance. Sections 4.1 and 4.2 are tightly connected. Section 4.3 states that reducing the first term of Eq. (5) in Section 4.2 is effective only while it exceeds the second term, and summarizes this as a practical takeaway on scalability limits. The claim of Section 4.3 builds on the equations in Section 4.2, which connects the two sections.

Following the comments of Reviewer bA72, we revised the key observation of Section 4.2 to the more direct “Same UTC ratio, different variance”. We hope this makes its contrast with Section 4.1 clearer.

all of Section 5 was poorly structured and I had no real clear understanding of why the specific choices of experiments were made and how the research questions laid out were motivated or justified.

We appreciate this feedback. The correspondence between each research question and Section 4 is as follows. Q1, Q2, and Q3 now state the corresponding section in the question itself. Q4 is not limited to the claims of a specific section and instead provides empirical support for practical guidelines.

  • Q1 (corresponding to Section 4.1): an empirical test of UTC invariance.
  • Q2 (corresponding to Section 4.2): measurement of each term in Eq. (5), and a test that the first term decreases as (E,R)(E, R) grows.
  • Q3 (corresponding to Section 4.2): a test of how the same UTC ratio with different variances affects performance.
  • Q4: a test of how small (E,R)(E, R) can be for the claims in Sections 4.1 and 4.2 to hold in practice. Its batch size results connect to Section 4.3.

At the beginning of Section 5, we added a paragraph explaining the motivation behind our research questions and the subsection that addresses each of them.

For the rationale behind our experimental design, we would also appreciate it if you could refer to our reply to Reviewer bA72.

The "Preliminary and Notation" section was unnecessary because the definitions provided in the section are repeated (and augmented) several times in following sections

Thank you for pointing this out. We believe this section has some value, as it defines the optimal policy and the cumulative reward in off-policy RL. On the other hand, the definition of the replay ratio, which is not used later, was unnecessary. Our original intention was to state explicitly that the replay ratio is a different concept from the UTD ratio, since the two are often confused. We have substantially shortened this section, keeping only the definition of off-policy RL and brief definitions of EE, LL, and the UTD ratio.

there is a lot of unnecessary and unused theoretical analysis

We believe that each part of the theoretical analysis supports our main argument. The longest derivation is in Appendix E, which derives the result of Section 4.2 and discusses the independence of samples in detail. In particular, we distinguish the correlation between individual samples (κ\kappa) from the correlation between the entire buffers at different steps (γK\gamma_K). If you could let us know which part you find unnecessary, we would be happy to explain its role in our analysis.

Revisions

Based on your review, we revised the manuscript as follows (changes shown in red).

  • We shortened Section 3 and removed the unused definition of the replay ratio and the descriptions that duplicate later sections.
  • We now explicitly state the motivation and corresponding subsection for each research question at the start of Section 5.
  • The research question in Section 5.3 now states the corresponding content of Section 4.2.
  • The discussion of batch size in Section 5.5 (previous Section 5.4) now states its connection to Section 4.3.

If any part of how the subsections of Section 4 connect, or how the experiments in Section 5 correspond to Section 4, remains particularly unclear, please let us know, and we will reflect it in future revisions.

Reviewer_wKnLOpenReview ↗

Improved structure, easier to follow the narrative and intended contributions!

Thank you for the revisions and additional experiments. The revised manuscript is more clearly structured, and the experiments are now much easier to follow. Section 5 explains the motivation for each research question and connects it to the analysis in Section 4. The added comparison of L-sweeps and R-sweeps also addresses an important concern about experimental design that I had. I am raising my score from "weak reject" but cannot advocate any stronger for acceptance. I do feel that additional development and justification of the experimental analyses are warranted before I'd be able to advocate for this paper to be published.

I still have major reservations about the experimental support for the proposed mechanisms. The main scale sweeps vary τ\tau through L, so the UTD ratio is varying. The new L-versus-R comparison helps, but it does not isolate the cause of the FastTD3 degradation at low τ\tau. Reviewer ba72’s follow-up supports this reservation. The added sweep showed substantial FastTD3 degradation at low UTC under the R-sweep. Reviewer ba72 also identified an unresolved confound between replay-buffer composition and update-frequency effects in the main sweeps. The variance analysis also measures snapshot proxies. It does not test the predicted multi-update variance with the effective horizon, temporal correlation, and FIFO corrections. The observed association between buffer variance and scale consistency therefore remains suggestive rather than mechanistically established.

The theory has always been relevant to the framing, but portions still feel extraneous to the primary narrative of the paper and the empirical analyses. The paper introduces a detailed variance decomposition, but the experiments do not directly validate several of its central assumptions or predicted terms. The revision improves the exposition and motivates the experiments more clearly, connecting the theoretical analysis where applicable. However, my central concern about the strength of the evidence linking the theoretical mechanisms to the observed learning behavior is still largely unresolved.

Thank you for recognizing that the revised manuscript is more clearly structured and easier to follow, and for reconsidering your assessment.

The main scale sweeps vary τ\tau through L, so the UTD ratio is varying. The new L-versus-R comparison helps, but it does not isolate the cause of the FastTD3 degradation at low τ\tau.

As Reviewer bA72 also pointed out, varying LL to change the UTC ratio also changes the UTD ratio. We fully agree that this confounding is an important consideration.

Since τ=Rσ\tau=R\sigma, changing the UTC ratio necessarily requires changing the buffer size RR, the UTD ratio σ\sigma, or both. In our additional experiments, shown in Fig. A and B in our response to Reviewer bA72 and Fig. 5 in the revised manuscript, performance degraded in some low UTC settings when RR was reduced while the UTD ratio was held fixed. The corresponding degradation was not observed when varying LL, which changes the UTD ratio. We therefore believe that the changes in the UTD ratio do not, by themselves, undermine the value of the experiments varying LL in Fig. 2 to 4.

We would also like to emphasize another point. The primary question in Fig. 2 to 4 is whether learning curves approach a common trajectory as (E,R)(E, R) increase proportionally at a fixed UTC ratio. The different UTC settings allow us to examine this scaling behavior across multiple temporal windows. This empirical question is distinct from identifying the cause of the FastTD3 degradation in the additional comparison.

We agree that the additional comparison does not isolate the mechanism responsible for this degradation. The result is consistent with the finite buffer contribution in Eq. (5), but does not establish it as the sole cause. Our choice of varying UTC through LL keeps the buffer size RR fixed within each subplot, avoiding the explicit change in the 1/R1/R factor introduced by varying RR.

The variance analysis also measures snapshot proxies. It does not test the predicted multi-update variance with the effective horizon, temporal correlation, and FIFO corrections. The observed association between buffer variance and scale consistency therefore remains suggestive rather than mechanistically established.

The paper introduces a detailed variance decomposition, but the experiments do not directly validate several of its central assumptions or predicted terms.

Thank you for highlighting the distinction between our snapshot measurements and the variance over multiple updates. We agree that we have not directly measured the effective horizon KK, the temporal correlation factor κ\kappa, or the FIFO correction γK\gamma_K, and our measurements do not directly validate the gradient variance over multiple updates. However, the buffer variance estimates in Fig. 7 and 12 are obtained by computing the mean gradient over the buffer samples within each environment lane and measuring the variability of these means across lanes. Assuming independence across environment lanes, this allows us to estimate Vlocal/(κR)V_{\mathrm{local}}/(\kappa R), incorporating temporal correlation within each lane without estimating κ\kappa separately. This snapshot estimate does not include γK\gamma_K, which accounts for the correlation of buffer errors across updates. Under the approximation in Appendix E, however, 2/3≤γK≤12/3\le\gamma_K\le1 for 1≤K≤τ1\le K\le\tau, so this correction reduces the first term by at most approximately 33%.

The minibatch variance measurements estimate the sampling variance at a single update and do not include the 1/K1/K reduction from averaging over multiple updates. Accordingly, the manuscript examines the scaling trends of the two measured quantities separately rather than directly comparing their absolute magnitudes.

Limitations remain, including local approximations such as a fixed model and the unknown effective averaging horizon. Despite these limitations, our analysis distinguishes temporal correlation within the buffer from correlation of buffer errors across updates. We also estimate the buffer and minibatch variances separately, with the buffer estimate accounting for temporal correlation within each environment lane. We would appreciate your consideration of these theoretical and empirical contributions.

Thank you again for your constructive feedback throughout the discussion, which has helped us improve the clarity of our paper.

We would like to add a clarification regarding the UTD ratios in our experiments, which we have also provided in our discussion with Reviewer bA72.

In Section 5.2 (Fig. 4), the UTD ratios range from 2−192^{-19} to 2−72^{-7} for E≥1,024E\ge1{,}024. In Section 5.4 (Fig. 5), the UTD ratios when varying LL range from 2−172^{-17} to 2−92^{-9}. Smaller values of EE yield higher UTD ratios in our sweeps. We highlight E≥1,024E\ge1{,}024 here because both FastTD3 (Seo et al., 2025) and FlashSAC (Kim et al., 2026) use 1,0241{,}024 parallel environments by default for MuJoCo Playground tasks.

Such small UTD ratios are common in existing implementations for massively parallel simulation. The official FlashSAC implementation uses 2−92^{-9} for both IsaacLab and MuJoCo Playground. The official FastTD3 implementation uses 2−102^{-10} for G1 and H1 in IsaacLab and 2−92^{-9} for T1 in MuJoCo Playground. Rybkin et al. (2025), an important prior study in this area, sweep UTD ratios from 2−162^{-16} to 2−102^{-10} on Isaac Gym.

Studies of overfitting often use a single environment and examine substantially higher UTD ratios. Rybkin et al. (2025) also report that "overfitting increases with UTD" in their DeepMind Control analysis, where the UTD ratios in their grid search range from 202^0 to 232^3. In the broader discussion on high UTD ratios, even higher ratios such as 2020 are also considered (Chen et al., 2021).

Thus, although the UTD ratio varies within our sweeps, its absolute value remains small in the settings specified above. We recognize, however, that the UTD ratio alone does not determine the extent of overfitting, which may also depend on factors such as batch size. We also recognize the importance of the confounding introduced by changes in the UTD ratio within our sweeps. We hope these values provide useful context for assessing concerns about overfitting and other effects associated with the UTD ratio.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.