acceptodds
Under review as a conference paper at ICLR 2027

LoGRA: Scaling LLM Reinforcement Learning with Low-Rank Gradient Sketches

Abstract

Reinforcement learning (RL) has greatly advanced the capabilities of large language models (LLMs), but its memory demands remain a barrier to broader adoption. We introduce LoGRA, an approach to RL post-training that reduces memory by retaining useful learning signals in low-rank gradient sketches. These compact representations support both model updates and efficient policy synchronization. To prevent overly large updates from disrupting learning, we complement gradient compression with predicted-KL step control, which estimates policy changes before applying each update and adjusts its magnitude accordingly. Across reasoning tasks, LoGRA reduces average training memory by up to 45.7% without sacrificing performance. It also enables stable training of a 27B-parameter model for over 1,100 steps on a single eight-GPU node, where dense Adam runs out of memory, making previously memory-infeasible RL training practical.

Primary area:
foundation or frontier models, including LLMs
Keywords:
RL Acceleration

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bdC6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LoGRA to reduce the memory requirements of reinforcement learning post-training for large language models. Its core idea is to accumulate random low-rank gradient sketches directly during backpropagation, merge the resulting updates into the model weights, and reuse the low-rank factors to synchronize the generation-side model. The authors also introduce predicted-KL step-size control, which estimates the policy change before applying an update and adjusts the update magnitude accordingly. Experiments cover 1.5B, 7B, and 27B models and demonstrate memory savings, reasonably comparable task performance, and the feasibility of sustained 27B training under a specified eight-GPU configuration.

Critical strengths
  1. The paper offers clear resource-efficiency value supported by evidence from actual training. In the 7B setting, average training memory drops from 31.82 GiB/GPU to 17.29 GiB/GPU, while peak accuracy and end-to-end update speed remain close to those of vanilla Adam. The main experiments include three random seeds. In addition, LoGRA enables 27B training under the specified eight-GPU configuration, which vanilla Adam cannot complete due to out-of-memory failures, and shows a sustained learning trajectory over more than 1,100 steps. These results indicate that the approach is practically useful; even without improving accuracy, lowering the resource barrier to training can itself constitute a meaningful contribution.
Critical weaknesses
  1. The incremental contribution relative to the closest prior work is not sufficiently validated. Flora already employs random projection, compressed gradient accumulation, and projection refreshing. The paper therefore needs to state more explicitly what additional capability is provided by direct sketch computation, policy synchronization, and step-size control, respectively. The current experiments mainly compare against vanilla Adam and LoRA, which is insufficient to determine whether an existing gradient-compression method integrated into the same RL framework would achieve similar gains. This affects the assessment of the size of the paper's contribution, rather than merely indicating the absence of one specific baseline.

  2. The additional benefit of predicted-KL has not been sufficiently established. In Figure 6, the baseline without predicted KL also remains stable; the small fixed-budget and annealed configurations improve late-stage average accuracy over this baseline by only 0.73 and 1.17 percentage points, respectively, with only two seeds per setting. The fixed-budget and annealed configurations also start from different budgets, which limits attribution of the annealing benefit. The authors' acknowledgment of these limitations is appropriate, but the current evidence does not establish a robust advantage of this component over reasonable step-size control. Moreover, the agreement between predicted and actual KL, as well as the estimation overhead, lacks direct verification. Since this component is a potentially important increment of the paper, its effectiveness substantially affects my assessment.

Questions for authors
  1. On positioning of the contribution: What are the key algorithmic and implementation differences between LoGRA and Flora, respectively? Which specific limitations of applying existing random gradient-compression methods within the current RL framework do these differences address? A controlled comparison at one representative scale, or a quantitative cost analysis grounded in implementation details, would help assess the added value; repeating experiments across all model scales is not necessary.

  2. On predicted-KL: Could the authors provide information on the agreement between predicted KL and the actual post-update KL, as well as the estimation overhead? Is there evidence that the benefit persists after reasonably tuning non-KL step-size control? In addition, given that the fixed-budget and annealed configurations have different initial values, how should the performance difference between them be interpreted?

4 comments
AuthorsOpenReview ↗

Official Reply to Reviewer bdC6 - Part 1

We sincerely appreciate your valuable time and insightful feedback! We address all concerns and questions in our detailed responses below. We will also integrate your valuable feedback in our revised version.

[Re: Weakness 1 & Question 1: Comparison with FLORA and Its Direct Application to RL]

Thank you for your valuable suggestions. We would like to clarify that LoGRA and FLORA target different domains. LoGRA focuses on RL post-training, where FLORA-in-RL is not directly applicable. Applying FLORA to the trainer alone does not automatically provide the RL-specific capabilities described below.

[1] Policy synchronization.

RL alternates between generating responses and updating the policy, requiring rollout engines to receive the trainer’s updates. FLORA does not provide a dedicated mechanism for fast weight synchronization; its gradient compression method alone does not address this communication cost.

FLORA first computes full gradients and then compresses them, whereas LoGRA computes gradient sketches directly from layer inputs during backpropagation. In this way, LoGRA only needs to send small update factors and random seeds instead of full weights, making policy synchronization more efficient.

Specifically, FLORA’s algorithm forms the full weight gradient G and then compresses it as S=GA^{⊤}. LoGRA instead projects the layer input x into Ax and combines it with the backward-pass signal to obtain S, without forming G. Optimizer processing and scaling produce a small update factor U. Subsequently, LoGRA sends only U and a random seed; rollout engines regenerate A and apply W← W-UA which avoid costly full-weight transfers.

We added a controlled 7B experiment on eight H200 GPUs, with four trainer GPUs and four rollout engines. Both methods apply identical updates. Synchronization time includes preparation, communication, reconstruction, and weight application.

MethodRank 64: payloadRank 64: timeRank 256: payloadRank 256: time
Full-weight synchronization, FLORA-in-RL14.185 GiB337.0 ± 2.3 ms14.185 GiB339.9 ± 2.6 ms
LoGRA (GPU-resident master weights)0.332 GiB229.3 ± 0.2 ms1.326 GiB284.1 ± 0.3 ms

Steady-state times are reported as the mean ± standard deviation of the medians from three runs, with 100 measured synchronizations per run. LoGRA reduces communication volume by 97.7% and 90.7%, yielding synchronization speedups of 1.47× and 1.20× at ranks 64 and 256, respectively.

[2] Step-size control.

Choosing a reliable step size in RL is difficult. Numerical settings such as the learning rate give users little intuition about how much the policy will actually change. So it is hard to set the number. A small parameter update can still cause a large change in policy behavior. We therefore introduce predicted-KL to estimate policy change before applying an update and shrink overly aggressive proposals.

We added a stress test to demonstrate this protection. Using Qwen2.5-Math-1.5B, rank 256, all runs start from the same trained checkpoint (~71% MATH-500 accuracy) and continue for 100 updates. We retain the tuned 1× non-KL baseline and enlarge its proposed update by 10× and 30×. At each enlarged size, we compare training with and without predicted-KL. Both controlled runs use the same fixed budget, 1.5×10⁻⁴, and only shrink proposed updates; measured post-update KL never controls training.

Proposed update sizePredicted-KL controlFinal accuracyConfirmed collapse update
1×Off71.15%None
10×Off1.45%70
10×On69.85%None
30×Off0.00%30
30×On70.05%None

Here, 1× denotes a proposed update-to-weight norm ratio of 6×10⁻⁵. Final accuracy uses 2,000 MATH-500 responses. Collapse requires a drop of at least 20 percentage points from initial accuracy for three consecutive evaluations; all runs complete 100 updates. Removing predicted-KL causes persistent collapse at both enlarged update sizes, whereas retaining it preserves accuracy near 70%. Predicted-KL is therefore a critical safeguard in these aggressive-update settings.

AuthorsOpenReview ↗

Official Reply to Reviewer bdC6 - Part 2

[Re: Weakness 2 & Question 2: the additional benefit of predicted-KL has not been sufficiently established]

Thank you for your valuable suggestions. We agree that Figure 6 alone does not establish the necessity of predicted-KL or isolate the benefit of annealing. We provide additional evidence and clarify the scope of our claims below.

[1] Protection against oversized updates.

As discussed in our response to Weakness 1 and Question 1, we added a single-seed stress test. We reproduce the results here for convenience.

Using Qwen2.5-Math-1.5B with rank 256, all runs start from the same trained checkpoint and continue for 100 updates. We retain the tuned 1× non-KL baseline and enlarge its proposed updates by 10× and 30×. Both controlled runs use the same fixed KL budget, 1.5×10⁻⁴, and only shrink proposals.

Proposed update sizePredicted-KL controlFinal accuracyConfirmed collapse update
1×Off71.15%None
10×Off1.45%70
10×On69.85%None
30×Off0.00%30
30×On70.05%None

Here, 1× denotes a proposed update-to-weight norm ratio of 6×10⁻⁵. Final accuracy uses 2,000 MATH-500 responses. Collapse requires a 20-percentage-point drop for three consecutive evaluations; all runs complete 100 updates.

The tuned 1× baseline remains stable. The additional benefit demonstrated here is protection against oversized proposals, rather than higher accuracy than well-tuned non-KL control. This is consistent with Figure 6.

[2] Prediction agreement and controller cost.

To examine KL prediction separately from gradient compression and training accuracy, we use an auxiliary dense-RL diagnostic with three RL objectives. For each measured update, we compare the prediction at the applied step size with post-update KL measured using separate auxiliary samples. The table compares these values over 400 updates per algorithm.

We report two types of metrics:

  • Actual / predicted KL: a ratio of 1 means agreement.
  • Pearson correlation: measures whether predicted KL tracks changes in actual KL across updates; values closer to 1 indicate stronger tracking.
RL algorithmUpdatesMean actual / predicted KLMedian actual / predicted KLPearson correlation
REINFORCE Zero4001.0321.0080.917
GRPO4001.0251.0110.934

On average, actual KL is 2.5–3.2% higher than predicted. Median ratios are within 1.1% of 1, and correlations exceed 0.91. These results show close aggregate agreement and strong tracking.

Thank you for the detailed rebuttal and additional experiments. The authors have adequately addressed my concerns and clarified the contributions of LoGRA and the role of predicted-KL. I am satisfied with the responses and will accordingly increase my rating from Weak Reject to Weak Accept.

AuthorsOpenReview ↗

Official Reply

Thank you for your thoughtful review and constructive feedback! We are glad that our responses and additional experiments have addressed your concerns. We sincerely appreciate your time and engagement, which have helped us improve the our work.

Reviewer_7z5FOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces LoGRA, a memory-efficient method for reinforcement learning post-training of large language models. For selected attention and MLP weight matrices, LoGRA directly accumulates randomized low-rank gradient sketches during backpropagation, avoiding full gradient buffers. It applies factorized updates to the original model weights and reuses these factors, together with projection seeds, to synchronize the rollout policy. The method also introduces predicted-KL step control, which estimates local policy divergence from directional changes in logits and scales updates according to a prescribed KL budget.

Experiments on DAPO-Math-7.5K with MATH-500 evaluation show that average training memory decreases from 9.18 to 7.18 GiB for the 1.5B model and from 31.82 to 17.29 GiB for the 7B model. Across three seeds, LoGRA achieves competitive peak accuracy relative to dense Adam. It also enables 27B training on a single eight-GPU node, using four GPUs for training and four for generation, where dense Adam runs out of memory during its first optimizer-state allocation. A separate single-seed Reasoning-Gym experiment sustains training beyond 1,100 logged steps, increasing the macro-averaged verifier score from 39.69% to 62.94%. Additional experiments examine projection rank, basis refreshing, projection distributions, LoRA comparisons, and KL-budget choices.

Critical strengths

The paper addresses a concrete memory bottleneck and demonstrates meaningful practical benefits. Table 1 and Figure 3 show substantial reductions in training memory for the 1.5B and 7B models under the same eight-GPU resource allocation. The 7B model maintains peak accuracy close to dense Adam, while the 27B experiment demonstrates a configuration in which LoGRA trains successfully and dense Adam cannot complete its first update. This is the strongest evidence supporting acceptance.

The scope of this result should remain precise: the reported 45.7% reduction concerns average training memory for the 7B model, rather than whole-node peak memory or a measured reduction for the 27B model.

Critical weaknesses

None

Questions for authors

What are the precise differences between LoGRA and Flora-style randomized gradient accumulation in backward computation, projection refreshing, factorized synchronization, and step-size control?

Does a closely related compression baseline implemented in the same RL framework preserve the claimed accuracy–memory advantage?

2 comments
AuthorsOpenReview ↗

Official Reply to reviewer 7z5F

We sincerely thank the reviewer for their time and insightful feedback, and for recognizing that our work addresses a concrete memory bottleneck and demonstrates meaningful practical benefits. We address all concerns and questions in detail below.

[Re: Question 1: Differences between LoGRA and FLORA]

Thank you for your valuable suggestions! We would like to clarify that LoGRA and FLORA target different domains. LoGRA focuses on RL post-training, where applying FLORA to the trainer alone does not automatically provide the RL-specific capabilities described below.

[1] Fast policy synchronization.

RL alternates between generating responses and updating the policy, requiring rollout engines to repeatedly receive the trainer’s updates. FLORA does not provide a dedicated mechanism for fast weight synchronization. Specifically, FLORA’s algorithm first forms the full weight gradient G, then compresses it as S=GA^{⊤}. LoGRA instead projects the layer input x into Ax and combines it with the backward-pass signal to obtain S directly, without forming G. This avoids costly full-weight transfers.

We added a controlled 7B experiment on eight H200 GPUs, with four trainer GPUs and four rollout engines and show the results below.

MethodRank 64: payloadRank 64: timeRank 256: payloadRank 256: time
Full-weight synchronization, FLORA-in-RL14.185 GiB337.0 ± 2.3 ms14.185 GiB339.9 ± 2.6 ms
LoGRA (GPU-resident master weights)0.332 GiB229.3 ± 0.2 ms1.326 GiB284.1 ± 0.3 ms

Compared with full-weight FLORA-in-RL synchronization, LoGRA reduces communication volume by 97.7% and 90.7%, yielding 1.47× and 1.20× synchronization speedups at ranks 64 and 256, respectively.

[2] Step-size control.

Choosing a reliable step size in RL is difficult because the learning rate gives little intuition about the resulting policy change. Predicted-KL estimates this change before applying an update. We test Qwen2.5-Math-1.5B, rank 256, for 100 updates from the same trained checkpoint (~71% MATH-500 accuracy). We retain the tuned 1× non-KL baseline and compare 10× and 30× proposals with and without predicted-KL. Both controlled runs use a fixed budget of 1.5×10⁻⁴ and only shrink proposals.

Proposed update sizePredicted-KL controlFinal accuracyConfirmed collapse update
1×Off71.15%None
10×Off1.45%70
10×On69.85%None
30×Off0.00%30
30×On70.05%None

Here, 1× denotes a proposed update-to-weight norm ratio of 6×10⁻⁵. Final accuracy uses 2,000 MATH-500 responses. Collapse requires a drop of at least 20 percentage points from initial accuracy for three consecutive evaluations; At both enlarged update sizes, training persistently collapses without predicted-KL, while retaining it preserves accuracy near 70%. Predicted-KL therefore serves as a critical safeguard in these aggressive-update settings.

[Re: Question 2: Controlled comparison in the same RL framework]

We try our best to implement FLORA’s Algorithm 1 in our RL framework. We compare Qwen2.5-Math-7B on eight H200 GPUs, SGD and step-size control. Each configuration completes 100 updates with three seeds. Using common SGD and control settings avoids attributing optimizer or controller differences to gradient compression.

RankMethodMATH-500 accuracy (%)Trainer peak allocated (GiB)Sync payload (GiB)Sync time (ms)
64FLORA-GA in RL (our implementation)68.25 ± 4.6521.9114.185337.0 ± 2.3
64LoGRA (GPU masters)72.25 ± 0.9222.140.332229.3 ± 0.2
256FLORA-GA in RL (our implementation)69.58 ± 4.4524.7214.185339.9 ± 2.6
256LoGRA (GPU masters)70.92 ± 3.2225.641.326284.1 ± 0.3

Accuracy is final MATH-500 pass@1, mean ± sample SD across three training seeds, using 2,000 responses per seed. Trainer memory is the maximum allocated memory across updates and trainer GPUs within each run, averaged across seeds. LoGRA achieves higher observed mean accuracy at both ranks. It also reduces payload by 97.7% / 90.7% and achieves synchronization speedups of 1.47× / 1.20× at ranks 64 / 256, respectively. As noted in our previous response, these synchronization gains are critical for RL.

Thank you for your answer, I have no problem now.

Reviewer_euRHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduce LoGRA, a method that uses low-rank gradient sketches to reduce the memory and communication costs of RL post-training, thereby lower the hardware barrier. LoGRA reduces the average memory cost of the 7B model by approximately 45.7% and enables the 27B model to complete over 1,100 training steps on a single eight-GPU node as the paper reports.

Critical strengths

1.By leveraging the associativity of matrix multiplication to directly compute compressed gradient contributions, LoGRA efficiently avoids the allocation of full gradient buffers. The most convincing result is that LoGRA succeeds training of a 27-billion-parameter model for over 1100 steps on a single eight-GPU node, where the dense Adam baseline runs out of memory.

2.This method applies low-rank representations to the weight updates on rollout policy, thereby jointly optimizing memory and communication overheads during RL training. Only the compressed update factors need to be transmitted to the rollout policy, which effectively and specifically reduces the weight update transmission cost.

Critical weaknesses

1.The paper mentioned that the dense baseline uses Adam with learning-rate annealing while LoGRA uses RowAdam, predicted-KL schedule and mismatch subtraction, the result proves that the full training method works while contributions of each module remains unclear.

2.Predicted-KL is indeed a useful method, but as author mentioned the no-control baseline also remains stable, and the separately collected annealed runs differ in their initial budget, so the results suggest a benefit from well-calibrated KL control without establishing that it is necessary for stability or that annealing alone explains the gain.

Questions for authors

see above

2 comments
AuthorsOpenReview ↗

Official Reply to Reviewer euRH

We sincerely appreciate your valuable time and insightful feedback! Thank you for highlighting the strengths of our work and providing so many detailed comments on our submission. We address each of your questions comprehensively in our responses below.

[Re: Weaknesses 1 & 2: Component Contributions and the Role of Predicted-KL]

We agree that the full-method comparison alone does not isolate each component. We therefore added a controlled experiment to examine predicted-KL separately and will clarify which implementation choices are claimed contributions.

Choosing a reliable step size in RL is difficult: the learning rate gives little intuition about how much the policy will change and RL is especially sensitive to it. Predicted-KL estimates this change before applying an update and shrinks overly aggressive proposals. We added a stress test using Qwen2.5-Math-1.5B, rank 256. All runs start from the same trained checkpoint (~71% MATH-500 accuracy) and continue for 100 updates. We retain the tuned 1× non-KL baseline and enlarge its proposed updates by 10× and 30×. Both controlled runs use the same budget, 1.5×10⁻⁴, and only shrink proposed updates.

Proposed update sizePredicted-KL controlFinal accuracyConfirmed collapse update
1×Off71.15%None
10×Off1.45%70
10×On69.85%None
30×Off0.00%30
30×On70.05%None

Here, 1× denotes a proposed update-to-weight norm ratio of 6×10⁻⁵. Collapse requires an accuracy drop of at least 20 percentage points for three consecutive evaluations. These results complement Figure 6: a tuned non-KL baseline can remain stable, while predicted-KL protects against oversized updates that otherwise cause persistent collapse. This supports its role as a safeguard against step-size misspecification. We also agree that the different initial budgets in Figure 6 prevent attributing the performance difference to annealing alone. We will revise this interpretation in our revised version.

We also kindly clarify Mismatch subtraction and RowAdam are supporting implementation choices rather than claimed contributions. Mismatch subtraction heuristically adjusts the available KL budget to account for rollout–trainer numerical mismatch, a problem studied by Qi et al. (2025) [1] and Zhong et al. (2026) [2]. These studies motivate handling mismatch, although our specific budget adjustment remains a heuristic. Similarly, RowAdam is a practical optimizer choice: Section 3.4.3 shows that LoGRA also trains effectively with SGD, which we use in the controlled experiment above. We will make it more clear in our revised version. Thanks for the valuable suggestion!

[1] Qi, P., Liu, Z., Zhou, X., Pang, T., Du, C., Lee, W. S., & Lin, M. (2025). Defeating the training-inference mismatch via fp16. arXiv preprint arXiv:2510.26788.

[2] Zhong, T., Ling, N., Pi, Y., Wei, Z., Yu, T., Fox, G., ... & Yu, X. (2026). Diagnosing Training Inference Mismatch in LLM Reinforcement Learning. arXiv preprint arXiv:2605.14220.

Thank you for taking the time to address my concerns. As most of my questions have been resolved, I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.