- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LoGRA to reduce the memory requirements of reinforcement learning post-training for large language models. Its core idea is to accumulate random low-rank gradient sketches directly during backpropagation, merge the resulting updates into the model weights, and reuse the low-rank factors to synchronize the generation-side model. The authors also introduce predicted-KL step-size control, which estimates the policy change before applying an update and adjusts the update magnitude accordingly. Experiments cover 1.5B, 7B, and 27B models and demonstrate memory savings, reasonably comparable task performance, and the feasibility of sustained 27B training under a specified eight-GPU configuration.
Critical strengths
- The paper offers clear resource-efficiency value supported by evidence from actual training. In the 7B setting, average training memory drops from 31.82 GiB/GPU to 17.29 GiB/GPU, while peak accuracy and end-to-end update speed remain close to those of vanilla Adam. The main experiments include three random seeds. In addition, LoGRA enables 27B training under the specified eight-GPU configuration, which vanilla Adam cannot complete due to out-of-memory failures, and shows a sustained learning trajectory over more than 1,100 steps. These results indicate that the approach is practically useful; even without improving accuracy, lowering the resource barrier to training can itself constitute a meaningful contribution.
Critical weaknesses
-
The incremental contribution relative to the closest prior work is not sufficiently validated. Flora already employs random projection, compressed gradient accumulation, and projection refreshing. The paper therefore needs to state more explicitly what additional capability is provided by direct sketch computation, policy synchronization, and step-size control, respectively. The current experiments mainly compare against vanilla Adam and LoRA, which is insufficient to determine whether an existing gradient-compression method integrated into the same RL framework would achieve similar gains. This affects the assessment of the size of the paper's contribution, rather than merely indicating the absence of one specific baseline.
-
The additional benefit of predicted-KL has not been sufficiently established. In Figure 6, the baseline without predicted KL also remains stable; the small fixed-budget and annealed configurations improve late-stage average accuracy over this baseline by only 0.73 and 1.17 percentage points, respectively, with only two seeds per setting. The fixed-budget and annealed configurations also start from different budgets, which limits attribution of the annealing benefit. The authors' acknowledgment of these limitations is appropriate, but the current evidence does not establish a robust advantage of this component over reasonable step-size control. Moreover, the agreement between predicted and actual KL, as well as the estimation overhead, lacks direct verification. Since this component is a potentially important increment of the paper, its effectiveness substantially affects my assessment.
Questions for authors
-
On positioning of the contribution: What are the key algorithmic and implementation differences between LoGRA and Flora, respectively? Which specific limitations of applying existing random gradient-compression methods within the current RL framework do these differences address? A controlled comparison at one representative scale, or a quantitative cost analysis grounded in implementation details, would help assess the added value; repeating experiments across all model scales is not necessary.
-
On predicted-KL: Could the authors provide information on the agreement between predicted KL and the actual post-update KL, as well as the estimation overhead? Is there evidence that the benefit persists after reasonably tuning non-KL step-size control? In addition, given that the fixed-budget and annealed configurations have different initial values, how should the performance difference between them be interpreted?
4 comments
Official Reply to Reviewer bdC6 - Part 2
[Re: Weakness 2 & Question 2: the additional benefit of predicted-KL has not been sufficiently established]
Thank you for your valuable suggestions. We agree that Figure 6 alone does not establish the necessity of predicted-KL or isolate the benefit of annealing. We provide additional evidence and clarify the scope of our claims below.
[1] Protection against oversized updates.
As discussed in our response to Weakness 1 and Question 1, we added a single-seed stress test. We reproduce the results here for convenience.
Using Qwen2.5-Math-1.5B with rank 256, all runs start from the same trained checkpoint and continue for 100 updates. We retain the tuned 1× non-KL baseline and enlarge its proposed updates by 10× and 30×. Both controlled runs use the same fixed KL budget, 1.5×10⁻⁴, and only shrink proposals.
| Proposed update size | Predicted-KL control | Final accuracy | Confirmed collapse update |
|---|---|---|---|
| 1× | Off | 71.15% | None |
| 10× | Off | 1.45% | 70 |
| 10× | On | 69.85% | None |
| 30× | Off | 0.00% | 30 |
| 30× | On | 70.05% | None |
Here, 1× denotes a proposed update-to-weight norm ratio of 6×10⁻⁵. Final accuracy uses 2,000 MATH-500 responses. Collapse requires a 20-percentage-point drop for three consecutive evaluations; all runs complete 100 updates.
The tuned 1× baseline remains stable. The additional benefit demonstrated here is protection against oversized proposals, rather than higher accuracy than well-tuned non-KL control. This is consistent with Figure 6.
[2] Prediction agreement and controller cost.
To examine KL prediction separately from gradient compression and training accuracy, we use an auxiliary dense-RL diagnostic with three RL objectives. For each measured update, we compare the prediction at the applied step size with post-update KL measured using separate auxiliary samples. The table compares these values over 400 updates per algorithm.
We report two types of metrics:
- Actual / predicted KL: a ratio of 1 means agreement.
- Pearson correlation: measures whether predicted KL tracks changes in actual KL across updates; values closer to 1 indicate stronger tracking.
| RL algorithm | Updates | Mean actual / predicted KL | Median actual / predicted KL | Pearson correlation |
|---|---|---|---|---|
| REINFORCE Zero | 400 | 1.032 | 1.008 | 0.917 |
| GRPO | 400 | 1.025 | 1.011 | 0.934 |
On average, actual KL is 2.5–3.2% higher than predicted. Median ratios are within 1.1% of 1, and correlations exceed 0.91. These results show close aggregate agreement and strong tracking.
Thank you for the detailed rebuttal and additional experiments. The authors have adequately addressed my concerns and clarified the contributions of LoGRA and the role of predicted-KL. I am satisfied with the responses and will accordingly increase my rating from Weak Reject to Weak Accept.
Official Reply
Thank you for your thoughtful review and constructive feedback! We are glad that our responses and additional experiments have addressed your concerns. We sincerely appreciate your time and engagement, which have helped us improve the our work.
Official Reply to Reviewer bdC6 - Part 1
We sincerely appreciate your valuable time and insightful feedback! We address all concerns and questions in our detailed responses below. We will also integrate your valuable feedback in our revised version.
[Re: Weakness 1 & Question 1: Comparison with FLORA and Its Direct Application to RL]
Thank you for your valuable suggestions. We would like to clarify that LoGRA and FLORA target different domains. LoGRA focuses on RL post-training, where FLORA-in-RL is not directly applicable. Applying FLORA to the trainer alone does not automatically provide the RL-specific capabilities described below.
[1] Policy synchronization.
RL alternates between generating responses and updating the policy, requiring rollout engines to receive the trainer’s updates. FLORA does not provide a dedicated mechanism for fast weight synchronization; its gradient compression method alone does not address this communication cost.
FLORA first computes full gradients and then compresses them, whereas LoGRA computes gradient sketches directly from layer inputs during backpropagation. In this way, LoGRA only needs to send small update factors and random seeds instead of full weights, making policy synchronization more efficient.
Specifically, FLORA’s algorithm forms the full weight gradient G and then compresses it as S=GA^{⊤}. LoGRA instead projects the layer input x into Ax and combines it with the backward-pass signal to obtain S, without forming G. Optimizer processing and scaling produce a small update factor U. Subsequently, LoGRA sends only U and a random seed; rollout engines regenerate A and apply W← W-UA which avoid costly full-weight transfers.
We added a controlled 7B experiment on eight H200 GPUs, with four trainer GPUs and four rollout engines. Both methods apply identical updates. Synchronization time includes preparation, communication, reconstruction, and weight application.
Steady-state times are reported as the mean ± standard deviation of the medians from three runs, with 100 measured synchronizations per run. LoGRA reduces communication volume by 97.7% and 90.7%, yielding synchronization speedups of 1.47× and 1.20× at ranks 64 and 256, respectively.
[2] Step-size control.
Choosing a reliable step size in RL is difficult. Numerical settings such as the learning rate give users little intuition about how much the policy will actually change. So it is hard to set the number. A small parameter update can still cause a large change in policy behavior. We therefore introduce predicted-KL to estimate policy change before applying an update and shrink overly aggressive proposals.
We added a stress test to demonstrate this protection. Using Qwen2.5-Math-1.5B, rank 256, all runs start from the same trained checkpoint (~71% MATH-500 accuracy) and continue for 100 updates. We retain the tuned 1× non-KL baseline and enlarge its proposed update by 10× and 30×. At each enlarged size, we compare training with and without predicted-KL. Both controlled runs use the same fixed budget, 1.5×10⁻⁴, and only shrink proposed updates; measured post-update KL never controls training.
Here, 1× denotes a proposed update-to-weight norm ratio of 6×10⁻⁵. Final accuracy uses 2,000 MATH-500 responses. Collapse requires a drop of at least 20 percentage points from initial accuracy for three consecutive evaluations; all runs complete 100 updates. Removing predicted-KL causes persistent collapse at both enlarged update sizes, whereas retaining it preserves accuracy near 70%. Predicted-KL is therefore a critical safeguard in these aggressive-update settings.