acceptodds
Under review as a conference paper at ICLR 2027

Understanding, Stabilizing and Accelerating High-Ratio N:M Sparse Training in Deep Reinforcement Learning

Abstract

Row-wise sparsity has emerged as a promising approach to accelerate matrix-matrix multiplications. However, the behavior of this type of sparsity remains under-explored in deep reinforcement learning (RL). Although magnitude-based sparsity implementations perform well at lower sparsity levels, they fail to maintain performance as sparsity increases. In this paper, we systematically investigate, stabilize, and accelerate high-ratio row-wise : sparse training in continuous-control RL. Under static 1:8 sparsity, randomly sampled initial mask topology has little predictive power for final return, whereas lower Fraction of Active Units (FAU) in the dense critic generally accompanies smaller sparse-dense return gaps. Under dynamic sparsity, we identify a three-stage actor saturation collapse and a mask-update shock in the critics, and counter them with scale-tuned initialization and transient replay-ratio increases. With increased width and a higher replay ratio, stabilized 1:8 agents achieve higher mean returns than same-width dense agents in 13 of 18 environment-buffer configurations. We further introduce time-multiplexed : training, which co-trains adjacent discrete patterns through shared parameters while keeping each subnetwork deployable at inference. On Walker2d, both subnetworks of a 3.5:8 schedule, trained with 12.5% fewer active weights than fixed 4:8, exceed independently trained 3:8 and 4:8 networks in mean return. Finally, our custom CPU sparse kernels provide 2.17-3.95 speedups for the forward pass of 1:8 layers and, combined with an approximate active-weight-only Adam optimizer, 1.34-1.83 end-to-end training speedups over an MKL-backed dense-kernel reference.

Primary area:
reinforcement learning
Keywords:
Semi-Structured Sparsity, Dynamic Sparse Training, Continuous Control, Efficient Reinforcement Learning, Training Acceleration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AUw3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The manuscript focuses on the under-exploration of the row-wise N:M network in the RL area. The motivation is clear but can be improved. The contributions are multi-folds, covering diagnosis and evaluation, stabilization techniques, and time-multiplexing sparsity processing. Experimental results are extensive and convincing, while the generality of the proposed solution can be improved.

Critical strengths
  1. Clear motivation
  2. Promising techniques although a bit incremental
  3. Extensive experiments
Critical weaknesses
  1. The proposed algorithm is mainly based on incremental innovation and evaluated with TD3.
  2. The evaluation in certain experimetns is not unified.
Questions for authors
  1. While Row-wise N:M sparsity is under-explored in RL, why do we want to use row-wise N:M sparsity in RL? The current described limitation seems to only focus on its less usage, but the rantionale behind this necessity and what benefit RLs could have by using this is not clear.

  2. Concerning the observation of smaller initialization, what happens when initialization is adjusted separately for actor and critic?

  3. The best return results use wider networks, while the end-to-end timing experiment uses width 256 and a different replay schedule. Can you report time-to-target-return and return-versus-wall-clock curves for the complete proposed configuration against tuned dense alternatives?

  4. Can the observed improvements be extended to other RL backbones?

1 comment

Thank you for the review. Your questions on the motivation, on the initialization, on wall-clock comparisons each led to an addition.

Q1: Why N:M sparsity for RL?

Because its regular structure turns sparsity into shorter training time, not merely fewer parameters. Larger networks are a main route to better returns in current deep RL, and Ma et al. (2025) showed that sparsity helps this scaling. But unstructured sparse networks are executed as masked dense ones and keep the dense cost. In our new experiments:

  • CPU: dynamic row-wise 1:8 TD3 trains 2.19–2.43× faster with PyTorch and 3.33–3.80× faster with native kernels than ordinary dense TD3 of the same width (one core, batch size 256).
  • GPU: static row-and-column 1:8 SAC/SimBa trains 3.2–3.7× faster at critic width 2048, with no significant difference in return.
  • Unstructured sparsity: ERK networks with the same number of active weights, run as masked dense layers, train at the speed and memory of the dense networks.

We strengthened this motivation in the Introduction, Related Work and Section 7.

W1/Q4: Novelty and other backbones

We agree that the paper builds on existing N:M sparsity and RNM-TD3, and we do not claim a fundamentally new RL algorithm. Its contributions are:

  1. Mechanistic diagnosis of capacity limits, actor saturation and mask-update shock.
  2. Controlled intervention analysis of initialization, width and replay ratio, including a new negative result: the timing of additional updates does not improve return beyond a constant schedule with the same number of updates.
  3. Time-multiplexed sparsity, which co-trains adjacent N:M patterns while every subnetwork stays deployable.
  4. Measured training acceleration against ordinary dense baselines on CPUs and GPUs.

New Section 7 extends the static result to SAC with SimBa residual networks; the stabilization of dynamic masks is not tested beyond TD3.

Q2: Separate actor and critic initialization

We ran this experiment (new Appendix K, Table 29): actor and critic initialization multipliers varied independently, six environments, 15 seeds per configuration.

Actor / critic scaleMean dense-normalized return
0.5 / 0.50.71
1 / 10.67
2 / 0.50.63
0.5 / 20.41
2 / 20.38

The critic's initialization scale has the stronger effect.

Q3: Return versus wall-clock time against tuned dense networks

  • CPU (Appendix E.4, Tables 15–18, Figures 11–12): two hours per configuration; sparse width 256 and 512 against dense width 256 and a dense network with the same number of parameters, at replay ratios 1, 2 and 4. At replay ratio 1, sparse width-256 reaches 4804 ± 943 on Ant and 138 ± 6 on Swimmer, against 2868 ± 1063 and 95 ± 24 for parameter-matched dense. A tuned dense width-256 agent reaches 5561 ± 625 on Ant and stays ahead; sparse width-512 (5657 ± 368) is not significantly different from it.
  • GPU (Figure 6, Appendix J.3): sparse width-2048 SAC reaches return 800 on HumanoidWalk after 2.2 hours, against 7.9 hours for dense of the same width. In the time dense needs for 0.5M interactions it completes 1.5M and reaches 966 ± 33 (HumanoidWalk) and 423 ± 69 (HumanoidRun), against 826 ± 143 and 192 ± 40.

W2: Unified evaluation

We agree. In Appendix E.4, all sparse and dense agents now train for two hours on one core with the same batch size (256), buffer (1M) and constant replay ratios (1, 2, 4), and Section 7 reports return and wall-clock time of the same runs. Still not unified: the speed ratios of Section 6 are measured at width 256 only, and Sections 3–5 keep their original settings, because each varies one factor against its own baseline.

Reviewer_o6GPOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the difficulties of training highly sparse neural networks for continuous control reinforcement learning. The authors investigate why aggressive sparsity can reduce performance and identify problems related to network capacity, actor saturation, and changes in sparse connections. They propose several improvements, including changes to initialization, temporary increases in replay updates, and wider sparse networks. The paper also introduces a training approach that alternates between different sparsity patterns while sharing model parameters. In addition, the authors develop specialized CPU implementations to speed up training. Experiments across several control environments show that the improved sparse models can achieve competitive or better rewards than dense models while reducing training time in the tested implementation.

Critical strengths

I think that the systematic investigation of sparse training failures is a major strength. The authors identify several causes of poor performance and propose specific changes to address them. This provides useful guidance for researchers working with sparse reinforcement learning models. Moreover, the paper measures actual training speed rather than relying only on theoretical reductions in computation. I appreciate the presentation of the measured end-to-end speedup over full one-million-step runs, with confidence intervals and similar results. The reported training-time improvements make the work more convincing from a practical perspective.

Critical weaknesses

I noticed that the experiments mainly use continuous control environments and one reinforcement learning algorithm. I think this limits understanding of whether the proposed improvements apply to other reinforcement learning methods and more complex tasks. Moreover, the speedup evaluation uses specialized CPU implementations and an approximate optimizer for active weights. It is therefore difficult to separate the benefits of sparse computation from those of the optimizer changes. I think that further comparisons are needed to establish the general efficiency advantage of the proposed approach. More specifically, regarding the experiments, only TD3 with small MLPs on six MuJoCo tasks is tested, and the speedup is measured only on a single CPU thread against a masked dense reference, while most RL training uses GPUs

Questions for authors

The authors should clarify how much of the reported training speedup comes from the sparse computational kernels and how much comes from the approximate optimizer. I am curious whether the same stabilization techniques are effective with other reinforcement learning algorithms. I would also like to know whether similar speedups can be expected on modern GPU hardware.

1 comment

Thank you for the review and for the questions on GPUs and on the optimizer; both led to new experiments.

W2/Q3: GPU acceleration

Yes at large width, for static masks, the gain shrinks at smaller widths. New Section 7 and Appendix J evaluate SAC with SimBa residual networks on one NVIDIA GH200, with the physics on the same GPU, using static row-and-column 1:8 masks that our kernels execute on Tensor Cores. We compare dense networks, 1:8 sparsity and unstructured ERK sparsity on HumanoidWalk, HumanoidRun and HopperHop.

Critic widthOne updateComplete training run
5122.1×1.2–1.3×
10243.8×1.9–2.2×
20485.1×3.2–3.7×

The complete run includes physics and evaluation, which sparsity does not change. Returns at width 2048:

TaskSparse 1:8Dense
HumanoidWalk815 ± 142826 ± 143
HumanoidRun201 ± 55192 ± 40
HopperHop353 ± 79347 ± 42

No statistically significant differences from dense were detected. In the time dense needs for 0.5M interactions, the sparse network completes 1.5M and reaches 966 ± 33 on HumanoidWalk and 423 ± 69 on HumanoidRun (no significant gain on HopperHop). Eight sparse agents with a critic of width 4096 train in 17.6 GiB, while eight dense agents of that width do not fit into the 96 GB of the GPU.

Unstructured ERK sparsity with the same number of active weights, run as masked dense layers, is not faster than dense and needs the same memory. This is why executable N:M structure matters.

W2/Q1: Sparse kernels versus optimizer

We removed the approximate optimizer and replaced the masked-dense reference (Section 6, Appendix E).

  • Reference. The comparison is now against ordinary dense TD3. Dense and sparse agents share all code except their hidden layers. On one CPU core, at width 256 and batch size 256, dynamic 1:8 TD3 trains 2.19–2.43× faster per environment step with PyTorch autograd and 3.33–3.80× faster with the native implementation.
  • Share of each component. Computed from the per-component profile (Table 14), forward and backward passes account for about 97–100% of the time saved and the Adam step for the rest.
  • Optimizer. The revision uses active-only Adam, which updates only active weights and their moments. Over 1M steps and ten seeds, standard Adam, the former approximation and active-only Adam give final returns without significant differences, except a higher return of active-only Adam on HalfCheetah (Table 9).
  • The table you appreciated. Table 9 of the original submission (1M-step runs with confidence intervals) measured against the masked reference, so we removed it together with that reference. The new timings are per update and per environment step (mean of three seeds, Tables 10–13); runs with 10–20 seeds are in the study at equal training time (Appendix E.4).
  • Equal training time. On a CPU, 1:8 TD3 beats a dense agent with the same number of parameters on Ant and Swimmer, but a tuned dense agent of the same width stays ahead on Ant (Appendix E.4).

W1/Q2: Generality beyond TD3

The new experiments add a second algorithm (SAC), a residual architecture (SimBa), much larger networks and a second benchmark suite, for static masks. Our GPU kernels support static masks only, so the actor-saturation analysis, mask-update shock, initialization remedies and time-multiplexing remain TD3 results. Discrete-action and pixel-based tasks remain untested.

Presentation

The paper now states which optimizer each experiment uses (Section 6, Appendix A.4), gives the replay-ratio schedule of every new experiment and distinguishes static from dynamic sparsity.

Reviewer_mzrsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates high-ratio N sparse training in TD3. It studies capacity limitations, actor saturation, and disruptions caused by mask updates, and evaluates initialization scaling, transient replay-ratio increases, and network widening as remedies. It also introduces time-multiplexed training of adjacent sparsity patterns. Experiments cover six MuJoCo tasks, with CPU kernels and an approximate active-only optimizer providing measured end-to-end acceleration.

Critical strengths
  1. The paper provides useful diagnostics of sparse RL training instability. In particular, distinguishing actor saturation from mask-update shock helps explain why dynamic sparsity can fail and motivates targeted interventions.

  2. The evaluation includes substantial replication across random seeds and measures actual execution time rather than relying solely on theoretical FLOP reductions. This strengthens the empirical contribution.

Critical weaknesses
  1. The practical advantage over simpler dense alternatives remains insufficiently established. Same-width comparisons show that sparse networks can retain performance, but do not fully answer whether sparsity is preferable to reducing dense-network width. A return-versus-wall-clock comparison against tuned dense networks across widths and replay ratios is important, especially because widening and additional updates are central to the proposed remedies.

  2. The acceleration results combine several changes and use a limited comparison setting. The reference already uses masked 1:8 networks evaluated through dense kernels, while the accelerated implementation also changes mask construction and Adam updates. Thus, the reported speedups do not isolate the benefit of sparse kernels or establish an overall advantage over ordinary dense training. Evaluation primarily on TD3 with small MLPs also limits the generality of the conclusions.

Questions for authors
  1. At matched actual training time or effective parameter count, how do the proposed sparse agents compare with smaller dense networks? Do the sparse configurations improve the return–time trade-off after tuning dense width and replay ratio?

  2. Can the end-to-end speedup be decomposed into contributions from sparse kernels, compiled mask construction, and approximate active-only Adam? How does the optimizer approximation affect returns and mask evolution?

  3. Does the transient replay-ratio schedule outperform a constant replay ratio with the same total number of optimizer updates? This comparison would help establish whether update timing specifically mitigates mask-update shock.

1 comment

Thank you for the detailed review. Your questions led to the largest changes of the revision: comparisons with dense networks at equal training time, a rebuilt speed evaluation, and one claim that we withdrew.

W1/Q1: Sparse versus tuned dense networks at equal training time

The answer depends on network size and task. In a new study, every configuration trains for two hours on one CPU core (Appendix E.4, Tables 15–18, Figures 11–12); we vary width, replay ratio and, on Ant, batch size. Return at replay ratio 1 and batch size 256 (native implementation, mean ± SD):

TaskSparse 1:8, width 256Dense, width 256Dense, equal parametersSparse 1:8, width 512
Ant4804 ± 9434852 ± 9342868 ± 10635657 ± 368
Hopper3171 ± 3423240 ± 5373187 ± 3083049 ± 847
Swimmer138 ± 695 ± 2495 ± 24116 ± 33
  • Against a dense network with the same number of parameters, the sparse agent is clearly better on Ant and Swimmer and not significantly different on Hopper.
  • After tuning batch size and replay ratio on Ant, the best dense width-256 agent reaches 5561 ± 625 and stays ahead of sparse width-256. Sparse width-512 reaches 5657 ± 368 in the same time.

On a GPU (new Section 7: SAC with SimBa networks, critic width 2048), static 1:8 masks train 3.2–3.7× faster. In the time dense needs for 0.5M interactions, the sparse network completes 1.5M and reaches 966 ± 33 on HumanoidWalk and 423 ± 69 on HumanoidRun, against 826 ± 143 and 192 ± 40; HopperHop shows no significant gain.

W2/Q2: What the speedup isolates

You are right on both points, and we replaced the evaluation (Section 6, Appendix E). The reference is now ordinary dense TD3 without masks; dense and sparse agents share all code except their hidden layers. On one CPU core, at width 256 and batch size 256, dynamic 1:8 TD3 trains

  • 2.19–2.43× faster per environment step with PyTorch autograd,
  • 3.33–3.80× faster with the native implementation.

Decomposition. Computed from the per-component profile (Table 14), forward and backward passes account for about 97–100% of the time saved and the Adam step for the rest; mask refreshes take less than 0.4% of a step.

Optimizer. The approximation is removed; the revision uses active-only Adam, which updates only active weights and their moments. Over 1M steps and ten seeds, standard Adam, the former approximation and active-only Adam give final returns without significant differences, except a higher return of active-only Adam on HalfCheetah (Table 9).

Q3: Transient versus constant replay ratio

Your concern was correct. We added the update-matched comparison (Appendix I, Tables 22–23). Interquartile-mean return across six environments, in percent of dense:

  • one update per step: 72.0
  • transient schedule: 83.5
  • constant schedule with the same number of updates: 87.5

The transient schedule shortens the half-recovery of the TD error after a mask update (17 instead of 85 environment steps), but the recovery needs about the same number of optimizer updates (139 and 147). The additional updates raise the return; their timing does not. We withdrew the claim and corrected the abstract, contributions, Section 4.1 and conclusions.

W2: Generality

Section 7 adds a second algorithm (SAC), a residual architecture (SimBa) and much larger networks on three DeepMind Control tasks, on a GPU. These results use static masks; the diagnosis and stabilization of dynamic sparsity are still shown for TD3 only.

Comments on the paper

1 comment

We thank all three reviewers for their constructive reviews. A revised PDF is uploaded. New and rewritten text is blue, and all section, table and figure numbers below refer to it. Detailed answers are in the individual threads.

Contribution. The paper asks three connected questions: (1) why high-ratio sparse RL training fails (capacity limits, actor saturation, mask-update shock); (2) which interventions actually help (initialization, width, replay ratio, time-multiplexed sparsity); (3) whether N:M sparsity reduces actual training cost, measured against ordinary dense training. The contribution is controlled intervention analysis and measured efficiency, rather than simply applying existing sparsity methods to RL.

Main changes

1. GPU acceleration and generality (new Section 7, Appendix J). SAC with SimBa residual networks and static row-and-column 1:8 masks on one NVIDIA GH200. At critic width 2048:

  • the complete training run, physics and evaluation included, is 3.2–3.7× faster;
  • in the time dense needs for 0.5M interactions, the sparse network completes 1.5M and reaches 966 ± 33 on HumanoidWalk and 423 ± 69 on HumanoidRun, against 826 ± 143 and 192 ± 40 (no significant gain on HopperHop);
  • eight sparse agents with a critic of width 4096 train in 17.6 GiB, while eight dense agents of that width do not fit into the 96 GB of the GPU.

2. Speed evaluation (Section 6, Appendix E). The reference is now ordinary dense TD3 without masks, and the approximate optimizer is removed. On one CPU core, dynamic 1:8 TD3 trains 2.19–2.43× faster with PyTorch autograd and 3.33–3.80× faster with a native C++ implementation. Computed from the per-component profile (Table 14), forward and backward passes account for about 97–100% of the time saved.

3. Dense comparisons at equal wall-clock time (Appendix E.4). After two hours on one CPU core, sparse 1:8 width-256 reaches 4804 ± 943 on Ant and 138 ± 6 on Swimmer, against 2868 ± 1063 and 95 ± 24 for a dense network with the same number of parameters. However, a tuned dense width-256 agent reaches 5561 ± 625 on Ant and is matched only by the sparse width-512 agent (5657 ± 368). All returns are for the same training time of two hours.

4. A corrected claim (Appendix I). Reviewer mzrs asked whether transient replay-ratio increases outperform a constant schedule with the same number of optimizer updates. They do not: interquartile-mean dense-normalized return is 83.5% for the transient schedule and 87.5% for the constant one. We withdrew the claim that the timing of the updates improves final return and corrected the abstract, contributions, Section 4.1 and conclusions.

5. Separate actor and critic initialization (new Appendix K). The critic's initialization scale has the stronger effect on return.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.