acceptodds
Under review as a conference paper at ICLR 2027

GRL: From GPU Underutilization to Faster Reinforcement Learning

Abstract

Compact networks in replay-based reinforcement learning (RL) often leave GPU capacity unused, yet increasing execution parallelism can also change the learning work needed to reach a target quality. We present GRL, a GPU execution workflow that preserves algorithm-defined learner-update structure while reorganizing collection, data delivery, and evaluation around it. Actor-side batching and cross-role overlap expose parallel work; GPU-resident handoff separates transfer completion from subsequent computation, minimizing synchronization overhead. Its learning-aware execution configurator screens candidates by throughput and compares learning quality through short trials, using a bounded search to limit selection overhead before training. Across eight environments with different complexity, GRL achieves end-to-end speedups up to over CPU-GPU asynchronous execution and over GPU-synchronous execution, measured across ten training runs sharing one configuration cost. Execution and configuration ablations show how throughput, learning progress, and selection overhead jointly shape first time-to-quality.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, GPU Computing, Automatic Configuration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yQHSOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GRL, a GPU execution framework designed to accelerate replay-based reinforcement learning with compact neural networks. It combines hierarchical GPU parallelism, asynchronous execution across actors, learners, and evaluators, and GPU-resident data exchange to improve hardware utilization. The authors further introduce a learning-aware configurator that selects execution parameters based on both throughput and short training trials. Experiments across eight environments using DQN, DDPG, and TD3 demonstrate improvements in first time-to-quality (FTQ)

Critical strengths
  1. The learning-aware configurator considers both execution throughput and learning progress, recognizing that higher throughput does not necessarily translate into faster convergence. This is a meaningful consideration for RL systems, where execution configurations can affect experience reuse and policy lag.

  2. The experiments demonstrate substantial FTQ improvements across several environments and learning algorithms. The execution ablations and GPU profiling results also provide useful evidence for the effectiveness of the proposed optimizations.

Critical weaknesses
  1. The methodological novelty appears limited. GPU-based environment batching, asynchronous actor-learner execution, and pipelined data exchange have been extensively explored in prior RL systems. GRL primarily integrates these established techniques, while the learning-aware configurator follows a relatively straightforward combination of throughput screening and short training trials. The paper could better distinguish its technical contributions from existing GPU-native RL and automatic configuration methods.

  2. The evaluation is limited to a single NVIDIA GH200 platform. Since the performance improvements depend heavily on GPU concurrency, memory architecture, and synchronization overhead, it remains unclear whether similar gains can be achieved on other GPU architectures or larger scale experiments.

  3. The effectiveness of the learning-aware configurator is not consistently demonstrated after accounting for selection overhead. In some settings, the configuration cost offsets the training-time improvements, and the benefits over maximum-parallelism configurations are relatively modest. This raises questions about when the additional configuration procedure is practically worthwhile.

Questions for authors
  1. Could the authors clarify the main technical novelty of GRL compared with existing GPU-native asynchronous RL frameworks? In particular, which mechanisms go beyond established actor-learner concurrency, GPU batching, and asynchronous data exchange?

  2. Could the authors provide a more detailed analysis of the break-even point for configuration overhead? For how many training runs must the selected configuration be reused before the configurator provides a net time benefit over simpler configuration strategies?

  3. How sensitive are the FTQ improvements to the choice of target reward? Would the relative performance remain consistent under lower or higher quality thresholds?

  4. How does GRL scale with increasing training workloads, such as larger neural networks, more parallel environments, and higher learner-update costs?

Reviewer_W5zQOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies GPU underutilization in replay-based reinforcement learning with small networks. The authors propose GRL, which leaves the algorithm's update logic unchanged and runs environment sampling, training, and evaluation in parallel on the GPU. Before training, a configurator selects the parallelism parameters. Experiments cover eight environments on a single GH200 and use the time to first reach a target return as the metric. GRL is compared against two execution modes, asynchronous CPU–GPU and synchronous GPU, and achieves speedups of up to 7.85× and 4.11× over them, respectively.

Critical strengths
  1. The paper measures time to reach a target return rather than throughput, so the metric accounts for both execution speed and learning quality. It also counts configuration overhead toward total time, which makes the conclusions fairly credible. Because the method does not modify the algorithm itself, it applies to a range of replay-based algorithms.
Critical weaknesses
  1. The novelty is limited. Batched environments, concurrent actors and learners, and replay-ratio control are already mature features of existing systems such as Isaac Gym and PQL, and this paper mainly combines these engineering techniques. Relative to prior work, its contributions are running evaluation in parallel with training plus implementation-level improvements such as double-buffered prefetching. Yet in the ablation, the pipelined handoff that includes these improvements yields only a 1.01–1.14× gain (Figure 4a).

  2. The baselines are weak. The authors implemented both baselines themselves and did not tune them separately. Open-source GPU training systems such as PQL and FastTD3, as well as GPU versions of the DM Control environments, are readily available. The paper neither compares against them nor uses those environments, and instead reimplements the environments. This limits how convincing the speedup results are.

  3. The measurement of GPU utilization is too coarse. The paper reports only the fraction of time kernels are active: 94.22% for Single-Stream and 99.53% for GRL. The two differ very little, yet their throughput differs by 1.52× (Figure 4). This suggests that the speedup comes mainly from more compute units working at the same time. However, the paper gives no finer-grained metrics such as SM occupancy, so the actual change in parallelism cannot be measured.

Questions for authors
  1. The current ablation disables double-buffered prefetching, non-blocking policy publishing, and parallel evaluation all at once, and Single-Stream also serializes all roles at the same time. Could the authors ablate each of these contributions separately and report how much each one contributes to throughput and time-to-target?

  2. Could the authors compare directly against systems such as PQL and FastTD3 on the same tasks, and report the SM occupancy of each method?

Reviewer_mS98OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the problem of low GPU utilization in reinforcement learning training with experience replay on small-scale networks. Taking time-to-first-quality (FTQ) — the time required to first reach the target quality — as the overall optimization objective, the paper combines multi-environment batching, parallel collection and training, data prefetching, and asynchronous evaluation into a GPU training workflow, while preserving the original algorithm's learner update structure. Building on this, the paper also designs a configuration selection method: it first filters candidate configurations by throughput, then compares the evaluation returns of policies from equal-duration short training runs, in order to select an appropriate number of actors, the number of environments per actor, and the experience replay ratio. Overall, the main contribution of the paper is to combine system execution optimization with configuration selection, jointly considering execution speed, learning effectiveness, and configuration overhead, so as to shorten the total time to reach the target quality. The work has some novelty, but overall it leans toward incremental innovation in system organization and configuration methodology.

Critical strengths
  1. It targets a well-defined system bottleneck and provides a complete implementation backed by experiments.

  2. It jointly considers execution speed, learning effectiveness, and configuration overhead, which is closer to the practical training objective.

Critical weaknesses
  1. The paper explains that fully measuring the FTQ of every candidate is too expensive, so it instead uses throughput filtering and equal-duration short training. However, "higher return within a short time" does not necessarily coincide with "reaching the target earlier in the end."

  2. The key workflow would be better explained through a concrete execution example. For instance, whether a Collect box in Figure 2 corresponds to a complete collection round. The text states the rule that "after the initial fill, a new policy is fetched before each collection round," but the figure does not clearly mark these boundaries.

  3. The workflow differences from related work are not sufficiently illustrated. Figures 1(a) and 2 lack a direct comparison with the closest prior work, making it hard to see which mechanisms GRL newly introduces.

  4. The advantage of the search strategy itself is not validated. In Section 6.4, Random samples only one configuration per seed, whereas GRL performs multiple probes and short training runs, making it difficult to tell whether the gains come from the search strategy or simply from a larger evaluation budget.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.