acceptodds
Under review as a conference paper at ICLR 2027

HiLoRe: What to Store, Compress, or Recompute for Efficient GRPO Training

20 views

Abstract

Group-relative policy optimization (GRPO) makes learner-side activations a major memory-computation bottleneck: gradient checkpointing reduces activation memory through recomputation, but fixed schedules can leave roughly 18 GB unused on a 48-GB GPU despite substantial recomputation overhead. Existing activation-management methods set state fidelity from execution cost, tensor properties, or generic compression sensitivity, without explicitly incorporating GRPO's analytic update structure into state-fidelity allocation. We formalize this dependence as policy-update exposure, linking the current GRPO loss coefficients to state-level approximation sensitivity. These coefficients are available before backward without an additional backward pass. We introduce **HiLoRe**, which allocates graph-attributed recovery units among gh-precision storage, w-precision compression, and deterministic computation using measured recovery utility and update-conditioned approximation risk. It combines high-precision storage and deterministic recomputation with low-precision recovery under a calibrated risk budget. Across five model-task settings with 2K responses and memory GC's per-GPU actor-update peak, HiLoRe's actor-update throughput gains reach 13.5% over GC and 7.9% over the fastest evaluated baseline, with paired mean downstream-score differences below 0.6 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, GRPO, Efficient Training

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VR3oOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

HiLoRe reduces recomputation during the GRPO actor update. For each graph-level recovery unit, it chooses whether to store the state in high precision (H), store a low-precision copy (L), or recompute it (R). These choices are guided by the exact per-token GRPO coefficients ωt=∂L/∂st\omega_t = \partial L/\partial s_t, which are available after the forward pass. These coefficients define a "policy-update exposure" score that bounds the gradient error introduced by compressing a unit (Thm. 1). A calibrated version of this score estimates the risk of L. A DP knapsack then selects H/L/R allocations within memory and risk budgets. With rank-16 LoRA on 3–8B models, 2K responses, and memory ≤1.10×\le 1.10\times GC, HiLoRe improves actor-update throughput by 10–13.5% over GC and by 4.9–7.9 points over the best baseline, while keeping downstream scores within 0.6 pp of GC.

Critical strengths

A useful GRPO-specific signal. The coefficients ωt\omega_t are exact and almost free to compute. They are also quite concentrated: the top 10% of slots account for 52–67% of the mass. As far as I know, previous activation-compression work has not used the update itself to decide state fidelity. I think this is the main idea of the paper, and it may also be usefull for other policy-gradient methods.

A careful evaluation. The authors compare with rematerialization, compression, and hybrid baselines, all using the same tuning budget. They test three models, three tasks, and 1K–4K responses. They also include a memory sweep, a fair packing comparison (Table 3), and gradient checks against GC. Since the timings include all online overhead, I think the speedup is real in the tested setting. Overall, the evalution is quite careful.

Critical weaknesses

The GRPO-specific contribution is limited. In Table 4, H/R allocation alone yields +6.14%, while static-risk H/L/R yields +8.31%. Update conditioning, the core novelty, adds about 1.9 points, bringing the gain to +10.16%. Most of the improvement comes from generic utility-driven scheduling, which is close in spirit to Adacc.

The theory is loosely connected to the method. Thm. 1 uses Jacobian norms, whereas the method uses activation magnitudes and a fitted constant κ\kappa. In practice, fidelity is controlled by an empirical tolerance (ϵg=0.015\epsilon_g = 0.015), and the risk predictor is only moderately accurate (Spearman 0.77).

The evaluation setting is narrow. The main results use LoRA, 3–8B models, and $\le$4K responses. The full-parameter run uses only one seed. The paper does not report end-to-end GRPO speedup, including rollout, leaving the practical impact of a 10–13% actor-update gain unclear.

Questions for authors
  1. What is the end-to-end wall-clock speedup, including rollout?
  2. Several baselines perform below GC, while HiLoRe uses the full 1.10×1.10\times memory budget. Did baseline tuning include configurations that use the same budget?
  3. Does the gain from update conditioning over static risk persist on Phi-3.5, Llama-3.1-8B, and under other memory budgets?
  4. H/R-only variants show a gradient error of about 0.003, although H and R should be exact. Could this be due to kernel nondeterminism? What is the error between two GC runs?
  5. The reported 54.8–73.6% zero-coefficient fraction includes padding slots. What is the fraction among valid response tokens?
  6. What are the one-time profiling and calibration costs, and how often must they be repeated?
9 comments
AuthorsOpenReview ↗

Response to Reviewer VR3o (Part 1 of 6)

Dear Reviewer VR3o,

Thank you for your careful review and for recognizing the value of update-aware recovery and the experimental evaluation. To facilitate your review and save time, we summarize our responses, supporting results, and manuscript revisions below.

ConcernResponseSupporting results and manuscript revisions
W1/Q3: Contribution and robustnessAdd controlled comparisons across models and budgets.Results: R1. Revised: Sec. 4.5 and App. D.7.
W2: Theory and predictor accuracyClarify the bound–surrogate connection and fidelity validation.Revised: Secs. 3.2–3.5 and App. C.2. Supporting results: Tables 12, 17.
W3: Evaluation scopeAdd 14B/8K evaluations and three-seed full-parameter results.Additional results: R2–R3 in this response.
Q1: End-to-end efficiencyCompare rollout-inclusive training time against GC and Adacc.Results: R4. Added: Sec. 4.2, Table 2. Protocols: Apps. D.1.4, D.7.
Q2: Baseline fairnessClarify shared memory limits and unused headroom.Existing protocol: App. D.1.3 and Table 9; clarified in this response.
Q4: H/R exactnessAdd repeatability and deterministic/execution-aligned controls.Results: R5. Revised: Sec. 4.5 and App. D.1.5.
Q5: Valid-token statisticsReport zero-contribution fractions excluding padding.Added: Sec. 4.4, alongside Fig. 5(a).
Q6: Preparation costsInclude preparation costs and clarify reuse conditions.Results: R12. Added: Sec. 4.2, Table 2. Clarified: Apps. D.1.4, D.7.

We sincerely hope that our responses, together with the theoretical clarifications and additional experimental results, help alleviate your concerns. If you find our responses helpful, we would be grateful if you would consider a more favorable assessment of our work. Detailed responses follow.

[continued below]

AuthorsOpenReview ↗

Response to Reviewer VR3o (Part 2 of 6)

W1/Q3: GRPO-Specific Contribution and Robustness

W1: Update conditioning adds only ~1.9 points; most gains come from generic scheduling.

Q3: Does its gain over static risk persist across Phi-3.5, Llama-3.1-8B and budgets?

Thank you for your careful review and for highlighting our GRPO-specific update signal as strength of the paper. We would like to clarify an important distinction: current-update adaptation is one component of our core contribution, and its 1.85-percentage-point increment does not measure the full GRPO-specific contribution. Static-risk already retains our GRPO-aware risk formulation, with risks fixed at their initialization averages. The controlled ablations below show that the gains from our GRPO-aware risk strategy over the generic-sensitivity alternative exceed Adacc’s overall gains over GC across the three evaluated models.

We use two complementary comparisons:

  • Generic-sensitivity vs. Full — value of GRPO-aware risk modeling. Generic-sensitivity replaces GRPO risk scoring with the ALAM-style estimator evaluated in Table 12, using the same calibration data and FP8 gradient-error targets. This comparison retains calibrated risk control and utility-driven scheduling while evaluating the complete GRPO-aware risk strategy.
  • Static-risk vs. Full — benefit of current-update adaptation. Static-risk retains each unit’s initialization-average GRPO risk; Full adapts that risk to the current update. This comparison isolates adaptation within GRPO-aware risk modeling.

All variants share recovery units, candidate-storage plans, the FP8 codec, action utilities, and allocator. Comparisons follow the original protocol and common tuning criteria, with all online estimation and scheduling costs included. Task scores summarize three independent training runs per method, with seeds paired across methods and each run evaluated after 60 updates.

Table R1. Measured comparisons of Generic-sensitivity, Static-risk, and Full HiLoRe across three models and additional Qwen memory budgets.

Model / budgetMethodPeak / $(B_{\mathrm{GC}}$) ↓Throughput gain over GC (%) ↑Mean grad. error ↓MATH500 (%) ↑GSM8K (%) ↑
Qwen2.5-3B / 1.10Generic-sensitivity1.0994.60 ± 0.370.014659.80 ± 0.6076.90 ± 0.19
Static-risk1.0988.31 ± 0.510.012860.13 ± 0.4277.43 ± 0.19
Full HiLoRe1.09610.16 ± 0.440.010460.73 ± 0.3177.84 ± 0.12
Phi-3.5-mini / 1.10Generic-sensitivity1.0996.10 ± 0.440.014243.73 ± 0.4285.42 ± 0.23
Static-risk1.0989.35 ± 0.430.012343.93 ± 0.3185.65 ± 0.23
Full HiLoRe1.09712.64 ± 0.470.009144.53 ± 0.5086.10 ± 0.19
Llama-3.1-8B / 1.10Generic-sensitivity1.0996.90 ± 0.440.014554.67 ± 0.6182.46 ± 0.23
Static-risk1.0999.70 ± 0.450.012754.93 ± 0.4282.71 ± 0.23
Full HiLoRe1.09613.47 ± 0.480.010055.53 ± 0.3183.09 ± 0.23
Qwen2.5-3B / 1.05Generic-sensitivity1.0491.80 ± 0.320.012160.00 ± 0.4077.18 ± 0.23
Static-risk1.0494.55 ± 0.420.010660.13 ± 0.3177.46 ± 0.19
Full HiLoRe1.0486.24 ± 0.430.008760.40 ± 0.4077.56 ± 0.15
Qwen2.5-3B / 1.15Generic-sensitivity1.1496.80 ± 0.370.014859.67 ± 0.7076.93 ± 0.27
Static-risk1.1489.95 ± 0.420.014059.93 ± 0.3177.38 ± 0.19
Full HiLoRe1.14712.63 ± 0.460.012260.33 ± 0.4277.53 ± 0.19

Table R1 supports two conclusions regarding W1 and Q3:

  • W1 — GRPO-specific contribution. At 1.10BGC1.10B_{\mathrm{GC}}, Full exceeds Generic-sensitivity by 5.56, 6.54, and 6.57 percentage points in throughput gain over GC on Qwen, Phi, and Llama, respectively, with all other recovery and allocation components shared. These increments exceed Adacc’s corresponding overall mean gains of 5.02%, 4.42%, and 6.38% over GC under the same memory ceiling (Table 1). Measured against the same GC baselines, these results demonstrate the substantial contribution of the GRPO-aware risk strategy.

  • Q3 — Robustness of current-update adaptation. Full exceeds Static-risk by 1.85/3.29/3.77 percentage points on Qwen/Phi/Llama at 1.10BGC1.10B_{\mathrm{GC}}, and by 1.69/2.68 points on Qwen at 1.05/1.15BGC1.05/1.15B_{\mathrm{GC}}. Thus, the adaptation benefit persists across the evaluated models and memory budgets.

The GRPO-specific contribution should also be assessed in terms of gradient fidelity and task performance. Across all five settings, Full achieves lower mean gradient error and higher mean MATH500 and GSM8K scores.

We have expanded Section 4.5 to distinguish the contribution of the GRPO-aware risk strategy from the additional benefit of current-update adaptation. The complete comparisons and evaluation protocol have been added to Appendix D.7.

[continued below]

AuthorsOpenReview ↗

Response to Reviewer VR3o (Part 3 of 6)

W2: Theory, Empirical Fidelity Control, and Predictor Accuracy

W2: The theory is loosely connected to the method. Thm. 1 uses Jacobian norms, whereas the method uses activation magnitudes and a fitted constant κ\kappa. In practice, fidelity is controlled by an empirical tolerance (ϵg=0.015\epsilon_g = 0.015), and the risk predictor is only moderately accurate (Spearman 0.77).

Thank you for raising this important point. HiLoRe’s risk proxy is constructed directly from the first-order gradient-error term derived from Theorem 1 (Eq. 52), preserving its GRPO-weighted sensitivity–distortion structure. It retains the exact GRPO coefficients and dependency support, approximates relative token sensitivity through activation-based weights, and calibrates the resulting features against measured single-unit gradient errors. The proxy score thus maintains a direct, close connection to the theoretical error term in both its components and structure.

Specifically, Eq. 52 gives

yu≤sucEueuL+Husu22c(euL)2,Eu=∑t∈J(u)∣ωt∣∥Gt,u∥op,y_u \le \frac{s_u}{c}\mathcal{E}_u e_u^L + \frac{H_u s_u^2}{2c}(e_u^L)^2,\qquad \mathcal{E}_u=\sum_{t\in\mathcal{J}(u)}\lvert\omega_t\rvert\lVert G_{t,u}\rVert_{\mathrm{op}},

where su=∥zu∥2+ϵs_u=\lVert z_u\rVert_2+\epsilon and cc is the stabilized reference-gradient norm. Our operational score retains this first-order exposure–distortion factorization:

ψu=κc(u)E^ueˉuL,E^u=∑t∈J(u)∣ωt∣αt,u\psi_u=\kappa_{c(u)}\widehat{\mathcal{E}}_u\bar{e}_u^L,\qquad \widehat{\mathcal{E}}_u=\sum_{t\in\mathcal{J}(u)}\lvert\omega_t\rvert\alpha_{t,u}.

Normalized activation RMS supplies mean-one relative sensitivity weights αt,u\alpha_{t,u}, while eˉuL\bar e_u^L estimates reconstruction distortion. Fitted by state type and layer, κc(u)\kappa_{c(u)} maps these features to observed single-unit errors, empirically accounting for effective sensitivity and normalization scales. Calibration remains fixed during training, while exposure tracks the current update.

Computing Gt,u=∂gt/∂zuG_{t,u}=\partial g_t/\partial z_u requires differentiating the parameter-gradient map; reviewer imZF likewise describes exact Jacobian computation as impossible at runtime. As you note, the GRPO coefficients are exact and almost free to compute. Our proxy preserves this exact signal and the theoretical scoring structure using forward reductions and weighted sums, without an additional online backward pass. Theory-guided sensitivity estimation also has precedents in activation compression: GACT uses paired gradient evaluations [1], while ALAM uses gradient-norm variance as a memory-efficient alternative to full-gradient variance [2]. Both support using tractable estimators whose accuracy is evaluated empirically.

HiLoRe’s component ablations directly test these approximations. In post-update diagnostics, removing activation weighting reduces Spearman correlation from 0.78 to 0.67, and removing susceptibility calibration reduces it to 0.74 (Table 13). Both improve the theory-guided score’s predictive value. The Spearman correlation of 0.77 meets common “strong correlation” conventions [3, 4], consistent with reviewer imZF’s observation that empirical alignment looks good. Beyond overall correlation, held-out single-unit interventions across three tasks yield Top-10% recall of 0.78–0.81 and AUROC of 0.86–0.89, exceeding the evaluated alternative signals (Table 12). These results directly support high-risk unit identification.

The theory-guided score governs online allocation through a surrogate risk budget, while ϵg\epsilon_g selects the operating point using development-set mean complete-update errors. Held-out complete-update checks assess the combined effects of proxy approximation and joint recovery, yielding a mean gradient error of 0.0104 and a maximum observed error of 0.0114 in the primary Qwen setting (Table 14). These checks complement single-unit diagnostics by assessing fidelity under actual training-update allocations. HiLoRe’s throughput advantage also persists across five shared tolerances from 1.0% to 2.0% (Table 17), showing that the efficiency benefit is not specific to ϵg=0.015\epsilon_g=0.015.

We have clarified this connection in Sections 3.2, 3.4–3.5 and Appendix C.2, showing how the bound guides the calibrated score and how predictor diagnostics and complete-update checks assess its predictive and operational effectiveness. We hope this clarifies the theoretical basis and empirical support for our allocation strategy.

[1] Liu et al. GACT: Activation Compressed Training for Generic Network Architectures. ICML 2022.

[2] Woo et al. ALAM: Averaged Low-Precision Activation for Memory-Efficient Training of Transformer Models. ICLR 2024.

[3] Schober et al. Correlation Coefficients: Appropriate Use and Interpretation. Anesthesia & Analgesia, 126(5): 1763–1768.

[4] Valentin et al. Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation. AAAI 2026.

[continued below]

Reviewer_Xn8bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents HiLoRe, a GRPO training system that dynamically decides whether learner-side recovery states should be stored in high precision, compressed in low precision, or recomputed. The key idea is to use GRPO’s per-token update coefficients to estimate how sensitive each recovery unit is to approximation, and then jointly optimize recovery time, memory use, and gradient fidelity under explicit budgets. The method is evaluated on 3B–8B models across math, code, and logic tasks, mainly with 4-GPU FSDP LoRA training. HiLoRe improves actor-update throughput by up to 13.5% over gradient checkpointing and 7.9% over the fastest evaluated baseline, while keeping gradient error and downstream-score differences small.

Critical strengths
  1. HiLoRe uses the current GRPO update coefficients to estimate which saved states are most sensitive to approximation, allowing recovery fidelity to adapt to the actual policy update.

  2. The paper compares against a broad set of recomputation and activation-compression baselines under shared memory and fidelity constraints, and includes online scheduling and compression overhead in the measured actor-update throughput.

Critical weaknesses
  1. Most experiments use rank-16 LoRA on 3B–8B models with four-rank FSDP, while the full-parameter evidence is limited to a 3B model with a short, single-seed training run. The paper therefore does not establish whether the gains persist for large-scale full-parameter GRPO or more complex distributed training with TP/PP/SP and multi-node execution.
Questions for authors
  1. Can the authors show whether HiLoRe’s throughput gains persist under more realistic large-scale distributed settings that include tensor, pipeline, or sequence parallelism and multi-node execution? The current evaluation uses four-rank FSDP with TP=PP=SP=1, so the interaction between recovery scheduling and communication-heavy training remains unclear.

  2. Can the authors provide stronger full-parameter GRPO results, preferably on larger models and over longer training runs? Most experiments use rank-16 LoRA on 3B–8B models, while the current full-parameter evidence is limited to a 3B model and a short single-seed run.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Xn8b (Part 1 of 2)

Dear Reviewer Xn8b,

Thank you for your careful review and positive assessment of our work. To facilitate your review, we summarize our responses, supporting results, and manuscript revisions below.

ConcernResponseResults and revisions
W1/Q1: Distributed scalingEvaluate 32B under FSDP/TP/TP+SP/PP and quantify decision overhead.R1–R2; App. D.8.1, Tables 21–22.
W1/Q2: Larger, longer full-parameter trainingCompare 7B full-parameter training over 500 rollout iterations against GC and Adacc.R3; Sec. 4.2; App. D.5.3, Table 16.

We sincerely hope these clarifications and additional results address your concerns and strengthen your support for our work. Detailed responses follow.


W1/Q1: Larger Models and Communication-Heavy Distributed Training

W1: Most experiments use 3B–8B models with four-rank FSDP, leaving gains on larger models under TP/PP/SP and multi-node training unestablished.

Q1: Do HiLoRe’s throughput gains persist under TP/PP/SP and multi-node execution, and how does recovery scheduling interact with communication-heavy training?

Thank you for raising these practical scaling questions. Our additional experiments show that HiLoRe’s throughput gains persist on a larger 32B model under tensor and pipeline parallelism. Recovery scheduling and its synchronization dependencies add only a small net runtime overhead, preserving these gains in the evaluated communication-heavy configurations.

1. Throughput gains persist at 32B under tensor and pipeline parallelism.

We test Qwen2.5-32B on an 8-L40S node: FSDP/TP/TP+SP/PP, 128 replay responses (106,888 valid prompt/response tokens), 4 sequences/microbatch, rank-16 LoRA, BF16 compute, FP32 persistent weights. TP+SP uses Megatron-style SP in the same 8-rank TP group, updating recovery utilities and risk calibration for sharded activations. Time/resources allow one seed.

We currently lack access to the GPU cluster resources needed for multi-node experiments and hope to explore these settings as time and resources permit.

Table R1. Actor-update performance of Qwen2.5-32B FSDP, TP, TP+SP, and PP on eight L40S GPUs.

Parallel layoutMethodActor time (s) ↓Throughput (valid tokens/s) ↑Peak memory/GPU (GiB) ↓Throughput gain
FSDP, 8 ranksGC110.91963.7742.87—
HiLoRe102.641,041.4344.108.06%
TP=8, DP=1, PP=1GC342.61311.9830.84—
HiLoRe306.69348.5233.6511.71%
TP=8 + Megatron-style SP, DP=1, PP=1GC331.86322.0924.65—
HiLoRe302.41353.4526.889.74%
PP=8, DP=1, TP=1; 1F1BGC125.82849.5238.41—
HiLoRe117.51909.6041.907.07%

HiLoRe retains 7.07%–11.71% actor-update throughput gains over matched GC at 32B, extending the original 3B–8B, four-rank FSDP evidence to tensor, pipeline, and sequence parallelism.

2. Recovery scheduling adds little net overhead in the evaluated communication-heavy configurations.

NCCL logs and GPU profiling help explain TP’s lower absolute throughput. L40S lacks NVLink; vLLM documentation recommends PP over TP for inference to reduce communication [1]. Training logs show shared-memory transport; a separate 64-response TP trace records 85.47 s of communication-kernel intervals, including communication-related waiting, in a 171.26 s GPU execution window. HiLoRe retains its Table R1 gain.

To isolate online decision overhead, we compare online execution with pre-supplied identical recovery actions, keeping inputs and candidate-storage plans fixed. The actor-time difference captures the net cost of risk collection, CPU allocation, transfers, and coordination, including host–device waiting on the execution path.

Table R2. Net runtime overhead of online recovery decisions on Qwen2.5-32B.

Parallel layoutPre-supplied actions (s)Online decisions (s)Added actor time (s)Added time / online total
FSDP, 8 ranks102.32102.640.320.31%
TP=8304.61306.692.080.68%
TP=8 + Megatron-style SP300.18302.412.230.74%
PP=8; 1F1B115.86117.511.651.40%

The complete online decision path accounts for only 0.31%–1.40% of actor-update time across FSDP, TP, TP+SP, and PP. With these costs included, HiLoRe retains the 7.07%–11.71% throughput gains in Table R1, showing that host-side scheduling and its synchronization dependencies do not become dominant bottlenecks in these configurations.

These results are included in Appendix D.8.1, Tables 21–22. We hope they address your concerns and will explore multi-node execution as resources permit. Thank you for your valuable suggestions.

[1] vLLM Documentation. Parallelism and Scaling. Section: “Distributed inference strategies for a single-model replica,” note: “Edge case: uneven GPU splits.”

[continued below]

AuthorsOpenReview ↗

Response to Reviewer Xn8b (Part 2 of 2)

W1/Q2: Full-Parameter GRPO on a Larger Model over Longer Training

Q2: Can the authors provide stronger full-parameter GRPO results, preferably on larger models and over longer training runs? Most experiments use rank-16 LoRA on 3B–8B models, while the current full-parameter evidence is limited to a 3B model and a short single-seed run.

Thank you for highlighting this need. The comparison extends full-parameter evaluation to Qwen2.5-Math-7B over 500 rollout iterations, comprising 4,000 optimizer steps. Across five checkpoints spanning early and late training, HiLoRe maintains higher actor-update throughput than both GC and Adacc, with task accuracy close to GC.

We follow One-Shot RLVR’s rollout schedule [1] on DSR-sub using eight L40S GPUs with FSDP FULL_SHARD, retaining our GRPO objective and optimizer settings. Task scores use greedy decoding at five checkpoints along each continuous run. Efficiency uses matched checkpoints and rollout batches under identical microbatch settings, including all optimizer steps and online costs. Gradient error is evaluated at matched states before clipping.

Table R3. Full-parameter GRPO on Qwen2.5-Math-7B / DSR-sub over 500 rollout iterations.

IterationMethodPeak MiB ↓Tok./s ↑Gain (%) ↑Mean grad. error ↓MATH500 (%) ↑GSM8K (%) ↑
0Shared initialization————50.8058.38
50GC36,7045,526.80——76.2086.81
Adacc39,6165,774.404.480.011476.4086.43
HiLoRe40,0646,237.5512.860.009075.8087.11
100GC36,7365,608.42——78.0089.46
Adacc39,6485,867.534.620.011677.6089.61
HiLoRe40,0966,321.2512.710.009278.4089.23
200GC36,8005,701.35——78.6090.07
Adacc39,7125,981.294.910.012078.8089.69
HiLoRe40,1606,447.0913.080.009578.2090.22
300GC36,8645,742.36——78.2089.92
Adacc39,7846,019.724.830.012277.8090.07
HiLoRe40,2326,485.4212.940.009778.8089.76
500GC36,9285,794.18——78.4090.22
Adacc39,8566,080.414.940.012578.4089.99
HiLoRe40,3526,557.2713.170.010178.2090.37

Across all five stages, HiLoRe improves actor-update throughput by 12.71–13.17% over GC and 7.73–8.02% over Adacc, with mean relative gradient error of 0.0090–0.0101, below the shared 0.015 tolerance, and peak memory within the common ceiling. All three methods improve rapidly during the first 100 iterations and subsequently fluctuate near their attained accuracy levels. HiLoRe stays within 0.60 percentage points of GC on MATH500 and 0.30 points on GSM8K throughout, reaching 78.20%/90.37% at iteration 500 versus GC’s 78.40%/90.22%.

This larger-model, longer-training comparison supports HiLoRe’s efficiency and quality preservation beyond the original 3B full-parameter setting. Its throughput advantage persists through early improvement and later training, without a widening accuracy gap from GC. We have added a summary to Section 4.2 and detailed results and protocols to Appendix D.5.3 (Table 16) of the revised manuscript. We hope these additions address your request for stronger full-parameter evidence.

[1] Wang et al. 2025. Reinforcement Learning for Reasoning in Large Language Models with One Training Example. NeurIPS 2025.


Thank you again for your constructive feedback. We have incorporated the additional experiments into the revised manuscript and hope our responses address your concerns. We would be grateful if these additions lead you to consider a more favorable assessment. Should you have further questions, we remain available for discussion.

Reviewer_Xn8bOpenReview ↗

Response

Thank you for the thorough response. My primary concerns have been adequately addressed, and I will maintain my positive rating.

AuthorsOpenReview ↗

Appreciation for Your Review

Dear Reviewer Xn8b,

Thank you for your follow-up and continued positive assessment, which is very encouraging to us. We are pleased that our responses have addressed your concerns. Your valuable suggestions motivated additional evaluations that strengthened the paper, and we sincerely appreciate the time and care you devoted to reviewing our work.

We sincerely hope the additional evidence further strengthens your confidence in our results and would be grateful if you find it warrants a more favorable assessment in any aspect of the work.

Best regards,

The Authors

Reviewer_imZFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HiLoRe (High/Low/Recompute), an adaptive activation-management framework tailored specifically to optimize the learner-side memory-computation bottleneck during Group Relative Policy Optimization (GRPO) training. While standard gradient checkpointing (GC) forces a rigid all-or-nothing recomputation schedule that often leaves valuable hardware memory headroom unexploited, HiLoRe dynamically decides whether to store activations in full precision (H), compress them into low precision (L), or completely recompute them (R).

Critical strengths

• Smart, Objective-Aware Design: Instead of managing precision based on generic tensor stats or execution costs, the paper beautifully exploits the unique mathematical structure of GRPO. Using early loss coefficients to predict gradient sensitivity is an elegant approach.

• Thorough Benchmarking: The authors didn't just compare their method against standard GC. They evaluated it against a strong lineup of recent memory-saving baselines (Adacc, AGoQ, INSTANT, Rockmate, etc.).

• Solid Results: The gains (up to 13.5% throughput improvement) are respectable. Crucially, the authors proved that these speedups don't hurt final model accuracy across math, code, and logic benchmarks.

• Validation at Scale: Testing full-parameter training on an A100 environment adds necessary real-world credibility beyond simple parameter-efficient (LoRA) settings.

Critical weaknesses

• High Operational Complexity: The engineering overhead is noticeable. Setting this up requires multi-step graph tracing, offline profiling to determine layer sensitivity, and running a knapsack solver on the host CPU during every microbatch. It is definitely not a simple "plug-and-play" utility.

• Evaluation Scale: While 3B to 8B models are standard for academic testing, modern post-training often uses much larger models (32B to 70B+). At that size, communication strategies like Tensor Parallelism (TP) dominate. The paper lacks proof that host-side scheduling won't cause sync bottlenecks when scaled across huge GPU clusters.

• Theoretical vs. Practical Gap: The underlying math (Theorem 1) relies on computing exact Jacobians, which is impossible at runtime. The framework substitutes this with a quick activation-magnitude proxy. While empirical alignment looks good, it means the framework relies heavily on a heuristic fallback.

Questions for authors

Overhead at Scale: How does the host CPU scheduling and dynamic programming overhead scale when running large models across complex Pipeline or Tensor Parallelism setups? Does host-device synchronization become a bottleneck?

Advanced Quantization: The framework uses basic unscaled FP8 for compression. Do you think introducing block-wise scaling or low-bit formats like INT4 would drastically change the current throughput-memory frontier?

5 comments
AuthorsOpenReview ↗

Response to Reviewer imZF (Part 1 of 3)

Dear Reviewer imZF,

Thank you for your careful review and positive assessment of our work. We are greatly encouraged by your recognition of GRPO-aware recovery, the baseline evaluation, and the preservation of training quality. To facilitate your review, we summarize our responses, supporting results, and manuscript revisions below.

ConcernResponseSupporting results and manuscript revisions
W1: Operational complexityClarify scripted setup, profile reuse, and preparation/runtime costs.Tables 2, 20; Sec. 4.2; Apps. D.1.4, D.7; repository README.
W2/Q1: Scale and scheduling overheadAdd 32B evaluations under FSDP/TP/PP and measure scheduling/synchronization overhead.Results: R1–R2. Added: App. D.8.1, Tables 21–22.
W3: Theoretical vs. practical gapClarify the proxy’s theoretical basis and calibration against gradient distortion.Clarified: Sec. 3.4, App. B.1. Theory: App. C.2, Eq. (52).
Q2: Advanced quantizationCompare FP8, block-scaled FP8, and INT4 under shared constraints.Results: R3. Added: App. D.8.2, Table 23.

We sincerely hope that our clarifications and additional experimental results help alleviate your concerns and further strengthen your support for our work. Detailed responses follow.


W1: Operational Complexity and Practical Benefit

W1 — High Operational Complexity: The engineering overhead is noticeable. Setting this up requires multi-step graph tracing, offline profiling to determine layer sensitivity, and running a knapsack solver on the host CPU during every microbatch. It is definitely not a simple "plug-and-play" utility.

Thank you for raising this practical concern. For the supported Qwen2.5-3B workflow, the supplied implementation handles graph tracing and recovery automatically, without requiring users to modify model or training code or write recovery rules. Our anonymous repository’s README provides a concise three-step guide: set paths and the risk budget, run initialization, and launch HiLoRe with the generated profile. LLM-assisted coding makes these routine command and configuration adjustments particularly straightforward.

Profiling and calibration are performed once and reused throughout training. Their additional preparation cost is recovered after 27 updates: preparation takes 61.4 minutes versus 3.2 minutes for GC, while each training update saves 2.20 minutes. Over 60 updates, including preparation, rollout, and other training-loop work, HiLoRe saves 73.8 minutes (3.76%) (Table 2).

Running the allocator every microbatch adds little overhead. Exposure/risk refresh and allocation together account for only 0.21% of actor-update time (Table 19). With transfers, synchronization, and scheduling included, actor-update time decreases from 23.85 to 21.65 minutes—a 9.22% reduction (Table 2).

We hope this clarifies why enabling HiLoRe in the supported workflow is simpler than its internal tracing, calibration, and scheduling procedures might suggest: the supplied scripts encapsulate these steps and keep setup straightforward. In the evaluated run, cumulative savings during training already account for online scheduling costs and exceed twice the additional preparation time, more than offsetting this one-time overhead.

[continued below]

AuthorsOpenReview ↗

Response to Reviewer imZF (Part 2 of 3)

W2/Q1: Larger Models, Scheduling Overhead, and Synchronization

W2 — Evaluation Scale: Academic tests commonly use 3B–8B models, but modern post-training often uses 32B–70B+, where TP communication dominates. Evidence that host scheduling avoids synchronization bottlenecks on large GPU clusters is missing.

Q1 — Overhead at Scale: How do host CPU scheduling and dynamic programming scale in large-model PP/TP setups? Does host–device synchronization become a bottleneck?

Thank you for raising these practical scaling questions. Our additional experiments show that HiLoRe remains effective when scaling to a larger 32B model, while host CPU scheduling and host–device synchronization do not become dominant bottlenecks in the evaluated parallel configurations. Including all online costs, throughput improves by 8.06% under FSDP, 11.71% under TP, and 7.07% under PP. Online recovery decisions add a net time overhead of only 0.31%–1.40% of actor-update time.

We extend evaluation to Qwen2.5-32B on one node with eight L40S GPUs, using one initialization seed given time and resource constraints. Comparisons share 128 replay responses totaling 106,888 valid prompt-and-response tokens, four sequences per microbatch, rank-16 LoRA, BF16 computation, and FP32 persistent weights. GC checkpoints all decoder layers; HiLoRe uses its GC baseline’s parallel layout.

Table R1. Actor-update performance of Qwen2.5-32B under FSDP, TP, and PP on eight L40S GPUs.

Parallel layoutMethodActor time (s) ↓Throughput (valid tokens/s) ↑Peak memory/GPU (GiB) ↓Throughput gain
FSDP, 8 ranksGC110.91963.7742.87—
HiLoRe102.641,041.4344.108.06%
TP=8, DP=1, PP=1GC342.61311.9830.84—
HiLoRe306.69348.5233.6511.71%
PP=8, DP=1, TP=1; 1F1BGC125.82849.5238.41—
HiLoRe117.51909.6041.907.07%

These results extend our efficiency evidence beyond the original 3B–8B models: HiLoRe continues to accelerate actor updates at 32B under all three tested parallel layouts. TP obtains the largest relative gain, while FSDP retains the highest absolute throughput. The potential benefits of recovery depend on the execution layout: avoiding recomputation can also eliminate associated communication in TP, while PP can benefit when recovery shortens the pipeline’s critical path.

To examine TP’s lower absolute throughput, we inspected NCCL logs, which show shared-memory transport. A separate 64-response TP trace shows communication-kernel intervals, including communication-related waiting, occupying 85.47 s of a 171.26 s GPU execution window, a substantial component. For hardware context, vLLM recommends PP over TP for inference on GPUs without NVLink, including L40S [1]. Our training-side observations motivate further evaluation with faster interconnects.

Each CPU dynamic-programming solve scales linearly with recovery units at fixed budget grids: O(UKBKδ)O(UK_BK_{\delta}), where UU counts units and KB,KδK_B,K_{\delta} are grid sizes (Appendix B.3). Total solver work also depends on allocation-call count. We directly measure actor-time impact, including statistics collection, transfers, and distributed coordination.

We compare online execution with pre-supplied identical recovery actions, holding inputs and candidate-storage plans fixed. The actor-time difference measures net online-decision overhead: risk collection, CPU allocation, transfers, coordination, and host–device waiting that extends actor execution.

Table R2. Net runtime overhead of online recovery decisions on Qwen2.5-32B.

Parallel layoutPre-supplied actions (s)Online decisions (s)Added actor time (s)Added time / online total
FSDP, 8 ranks102.32102.640.320.31%
TP=8304.61306.692.080.68%
PP=8; 1F1B115.86117.511.651.40%

The net overhead remains small across all three layouts. TP has the largest absolute increase, whereas PP has the largest relative increase. Compared with TP, PP adds less time in absolute terms but has a substantially shorter total update time.

This includes obtaining GPU statistics and returning CPU decisions. Together, the small net overhead and Table R1’s throughput gains show that host scheduling and synchronization are not dominant bottlenecks for 32B in these tested configurations.

We hope these experiments address your concerns about larger models and HiLoRe’s scheduling and synchronization costs. As resources permit, we would like to explore NVLink-connected systems and larger multi-node clusters. Thank you for prompting this closer investigation of practical scaling.

[1] vLLM Documentation. Parallelism and Scaling. Section: “Distributed inference strategies for a single-model replica,” note: “Edge case: uneven GPU splits.”

[continued below]

AuthorsOpenReview ↗

Response to Reviewer imZF (Part 3 of 3)

W3: Theoretical Connection and Calibrated Risk Estimation

W3 — Theoretical vs. Practical Gap: Theorem 1 requires exact Jacobians, infeasible at runtime. The method uses an activation-magnitude proxy; despite good empirical alignment, it relies heavily on a heuristic.

Thank you for raising this concern. While the practical score is approximate, it is grounded in the error structure of Theorem 1 and calibrated against actual gradient distortion, rather than relying on activation magnitude alone. Specifically, the first-order term in Eq. 52 motivates combining current-update exposure with reconstruction error. HiLoRe preserves the exact GRPO coefficients and graph-derived dependency support, while using activation features and state/layer-specific calibration fitted to measured gradient errors. This retains the analytical structure of the error model while enabling practical risk estimation without an additional online backward pass.

Activation magnitude supplies information about the scale of compression perturbations. Under normal-range FP8 round-to-nearest conversion without overflow or underflow, absolute rounding error is bounded in proportion to activation magnitude. Normalized activation RMS therefore preserves relative token magnitudes and, when relative rounding errors are comparable, provides information about how perturbation magnitudes vary across tokens. Its role is to supply a useful distortion-related feature; the relationship to gradient error is fitted through calibration.

The resulting score is

ψu=κc(u)eˉuL∑t∈J(u)∣ωt∣αt,u,\psi_u=\kappa_{c(u)}\bar{e}_u^L\sum_{t\in\mathcal{J}(u)}\lvert\omega_t\rvert\alpha_{t,u},

where eˉuL\bar{e}_u^L is the mean relative reconstruction error over calibration microbatches, αt,u\alpha_{t,u} captures relative activation magnitude, and ∣ωt∣\lvert\omega_t\rvert supplies the exact current-update weighting. Calibration fits κc(u)\kappa_{c(u)} by state type and layer using matched single-unit relative gradient errors, relating these update-weighted reconstruction features to observed gradient distortion and capturing effective sensitivity and normalization scales.

We hope this clarifies the connection: the theory guides the structure of the estimator, calibration relates its inexpensive features to actual gradient distortion, and held-out evaluation supports the usefulness of the resulting approximation. Together, these provide a principled and practical basis for update-aware allocation without evaluating exact Jacobians online.


Q2: Block-Wise Scaling and INT4 on the Throughput–Memory Frontier

Q2 — Advanced Quantization: The framework uses basic unscaled FP8 for compression. Do you think introducing block-wise scaling or low-bit formats like INT4 would drastically change the current throughput-memory frontier?

Thank you for this insightful and constructive question. At our primary memory budget, these alternatives improve the trade-off modestly: throughput increases by 1.04–1.31% relative to unscaled FP8, with task performance remaining close to GC.

Our original design uses unscaled E4M3FN [1] for its simple conversion without scale estimation or metadata. Your suggestion motivates a comparison with block-scaled FP8, which can improve range utilization, and INT4, which further reduces the quantized payload. Their net benefits depend on reconstruction fidelity and encoding/reconstruction costs, which HiLoRe incorporates through codec-specific calibration and utility profiling.

All comparisons follow the paper’s original Qwen2.5-3B protocol at 1.10BGC1.10B_{\mathrm{GC}}, with the same search allowance and memory and quality constraints. We reprofile and recalibrate each codec, allowing recovery assignments to adapt; throughput includes all online costs.

Table R3. Codec comparison within HiLoRe at 1.10BGC1.10B_{\mathrm{GC}}.

CodecPeak memory (MiB)Gain over GC (%)Mean grad. errorMATH500 (%)GSM8K (%)
Unscaled FP831,64910.16 ± 0.440.010460.73 ± 0.3177.84 ± 0.12
Block-scaled FP831,61011.30 ± 0.470.009160.60 ± 0.4077.79 ± 0.15
Block-scaled INT431,59011.60 ± 0.570.013860.27 ± 0.4277.51 ± 0.19

Block-scaled FP8 emerges as a promising direction: it achieves nearly the same throughput as INT4 with lower complete-update gradient error. We are extending this evaluation to other models to assess whether the benefit persists. Thank you for this valuable suggestion, which helps us further improve HiLoRe.

[1] Micikevicius et al. 2022. FP8 Formats for Deep Learning. arXiv:2209.05433.


Thank you again for your careful review and constructive feedback. We have incorporated the clarifications and additional results discussed above into the latest manuscript. We hope these revisions and responses address your concerns, and we would be happy to answer any further questions.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on Our Responses to All Reviewers

Dear Reviewers,

Thank you for your valuable feedback, which has helped improve the rigor and clarity of our paper. We greatly value the opportunity to discuss our work with you and would welcome your thoughts on our responses and revised manuscript as the discussion continues.

We have provided additional experiments and clarifications, with concise summaries to facilitate review. We would appreciate knowing whether these address your concerns and would be grateful if they support a more favorable assessment of our work. Please let us know if any questions remain; we would be happy to clarify further.

Best regards,

The Authors

AuthorsOpenReview ↗

A Gentle Follow-up

Dear Reviewers,

Thank you for your valuable feedback. We have carefully considered your comments and devoted substantial effort to our responses and additional experiments. Before the discussion closes, we would greatly appreciate any further feedback and would be grateful if the new evidence supports a more favorable assessment. We remain available to clarify any remaining questions.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.