acceptodds
Under review as a conference paper at ICLR 2027

MaskLift: Exact Mask Replay for RL under KV-Cache Compression

Abstract

Online RL post-training spends most of its memory budget on the rollout phase, where the Key-Value (KV) cache of long prompts is the part of the footprint that grows with context and batch. Compressing that cache relieves the pressure but breaks the assumption the policy gradient rests on: trajectories are sampled under a compressed context while the learner scores them under the full one, so the update is off-policy by construction. Remedies for it correct the resulting estimator; we remove what it estimates. MaskLift records the discrete retention mask the rollout compressor produced for each sequence and replays it in the learner's attention, so the sparse/dense importance ratio is identically 1 rather than reweighted or rejected. That identity is exact at the parameters that produced the trajectory, and it is a property of the sparse policy; the dense policy deployment uses inherits the improvement only through a second term, which detaches the RL-improved sparse pass and fits the dense pass to it. On GSM8K under our retention rule, where every uncorrected-learner run finishes below the untrained checkpoint's 40.7, MaskLift reaches 72.8 over its surviving runs against 72.3 for a dense-cache learner that never compressed anything, collapsing on 1 of 5 runs against 0 of 5. It tracks that learner on three further datasets, and we map where the mismatch is severe and where it costs nothing. Qwen3-1.7B, contexts up to 4K; the memory saving is confined to the rollout-decode phase.

Primary area:
reinforcement learning
Keywords:
reinforcement learning for LLMs, KV cache compression, policy gradient, rollout-learner mismatch, importance sampling, GRPO, efficient post-training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_iCYhOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The method optimizes the sparse-attention policy using a standard on-policy RL loss by replaying the rollout’s retention mask in the learner. An additional KL loss aligns the full-attention policy with the sparse-attention policy, which shares the same parameters and serves as a detached teacher. The ultimate goal is deployment with full attention.

Critical strengths
  1. Well-motivated objective. Mask replay enables on-policy optimization of the sparse policy, while the KL term addresses the gap to full-attention deployment.

  2. Transparent reporting. The paper discloses failed runs and reports both conditional and all-run statistics, helping readers assess the method’s benefits and limitations.

Critical weaknesses
  1. Sparse-RL baseline fidelity. In the supplementary code (code/masklift/losses.py, lines 149–158 and 212), the loss is normalized by the number of accepted trajectories rather than the original group size specified in Sparse-RL. In my independent three-seed control, correcting only this denominator improved GSM8K accuracy by 6.1–7.6 percentage points. This weakens the reported comparison with Sparse-RL.

  2. Insufficient evidence for the performance–cost tradeoff. The paper does not report held-out performance against wall-clock time, including learner and transfer costs. Figure 2 omits intermediate-checkpoint performance and emphasizes non-collapsed runs. On GSM8K, including all runs reduces MaskLift’s mean accuracy from 72.8% to 61.6%, compared with 72.3% for Dense GRPO. The results therefore do not establish a favorable overall performance–cost tradeoff or show whether the 200-step conclusions persist with longer training.

Questions for authors

Please address the concerns raised in the weaknesses above, particularly the Sparse-RL implementation and the end-to-end performance–cost comparison.

Reviewer_xxVBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MaskLift to address the rollout–learner policy mismatch caused by KV cache compression. The method records the KV retention masks actually used during generation and exactly replays them during training, so that the learner and the rollout compute policy probabilities under identical information conditions. A sparse-to-dense KL loss is then added to encourage transfer from the sparse policy to the full-context policy. On four tasks with Qwen3-1.7B, experiments show that the method mitigates compression-induced performance degradation and, in some settings, approaches the performance of Dense GRPO.

Critical strengths

The paper tackles a practically important question—how KV cache compression during RL post training can create a rollout “learner policy mismatch” and its controlled experiments show that this mismatch can cause substantial performance degradation, making the analysis useful for assessing compression in this setting. The method is cleanly designed and its core mechanism is sensible: replaying the actual retention masks from rollout ensures that the learner computes policy probabilities under the same information conditions, while the sparse-to-dense KL connects the training and deployment branches. The exact-replay claim holds under clearly stated conditions, and the authors account for key implementation details such as prefill, positional encoding, and layer- and head-wise masking. On the empirical side, four tasks, multiple seeds, and supplementary experiments with a real SnapKV scorer suggest that the gains are not merely an artifact of a particular positional retention rule, and the disclosure of failed runs and implementation limitations helps readers gauge how far the findings generalize.

Critical weaknesses
  1. End-to-end efficiency gains have not yet been demonstrated. The current memory savings occur only during decoding. The peak memory and maximum batch size for the full run are not improved, and the overall cost of the additional dense forward pass has not been adequately evaluated. We recommend that, under longer contexts, the authors report full training peak memory, maximum feasible batch size, and the time and GPU-hours required to reach the same task quality.

  2. Key performance comparisons are confounded by evaluation protocols and failed runs. Some methods use different generation limits and hardware. The conclusion that MaskLift is close to Dense GRPO comes mainly from successful runs. When all runs are included, a clear gap remains. We recommend standardizing hardware, evaluation scripts, and generation budgets, and running paired experiments with a pre-specified number of seeds while reporting performance over all runs, failure rates, and the magnitude of failures.

  3. The mechanism behind the transfer loss and whether it is actually necessary have not been adequately validated. The existing ablations only show that adding KL helps. They cannot distinguish targeted knowledge transfer from generic consistency regularization, nor do they show that sparse→dense is better than the reverse design. We recommend comparing no KL, sparse→dense, dense→sparse, and bidirectional consistency under the real SnapKV setting, while evaluating the performance of both the sparse and dense branches.

Questions for authors

see above.

Reviewer_XZSCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes MaskLift, a framework for reinforcement learning under KV-cache compression. The authors identify a policy mismatch between sparse rollouts generated with compressed KV caches and dense learner updates, which can lead to unstable RL training. To address this issue, MaskLift records and replays the exact KV retention mask during the learner's forward pass, eliminating the compression-induced importance ratio mismatch. Since the resulting policy gradient optimizes the sparse policy, the authors further introduce a KL-based transfer loss to align the dense policy with the improved sparse policy. Experiments on Qwen3-1.7B across four datasets demonstrate that MaskLift can mitigate training failures caused by KV compression while maintaining performance comparable to dense GRPO in some settings.

Critical strengths
  1. The problem is well-motivated and practically relevant. The policy mismatch introduced by KV-cache compression during RL training is an interesting issue worth investigating.

  2. The proposed method is simple and intuitive. Replaying the exact retention mask provides a straightforward way to address the policy mismatch without relying on importance sampling or rejection sampling.

  3. The experimental analysis is relatively comprehensive in terms of training diagnostics. The authors investigate policy mismatch, training collapse, retention ratios, and the effect of the KL transfer term.

  4. The paper is generally well-written, and the motivation and methodology are easy to follow.

Critical weaknesses
  1. The methodological novelty seems somewhat limited. The main components, including mask replay and KL-based distillation, are closely related to existing techniques. The authors also acknowledge that record-and-replay has been explored in R3 and QaRL.

  2. The experiments mainly rely on a simple position-based KV-cache eviction strategy. Although the authors provide additional experiments with SnapKV, several widely used KV-cache compression methods, such as H2O, are not evaluated.

  3. The experimental evaluation is limited in terms of model diversity and scale. All end-to-end RL experiments are conducted on Qwen3-1.7B. Given that KV-cache memory becomes increasingly important for larger models and longer contexts, it would be helpful to evaluate MaskLift on larger models to demonstrate its scalability and generalizability.

  4. No end2end speedup results reported. It would be necessary to understand how the proposed method help improve the rl efficiency.

Questions for authors

listed in weakness.

Comments on the paper

1 comment
Reviewer_iCYhOpenReview ↗

Independent reproduction under two prompt settings

Thank you for the updated code.

TL;DR: In my independent experiments, MaskLift performs worse on average than corrected Sparse-RL under a more reasonable prompt setting for GSM8K; hence, my score remains unchanged.

The two prompt settings are shown schematically below; the worked examples are abbreviated.

# Authors' prompt
Solve the math problem step by step; put the answer in boxed{}.
Example 1:
Problem: <Natalia clips question>
Solution: <worked solution; boxed{72}>
Now solve this problem:
Problem: <target question>
<generation starts here>

# My more reasonable prompt
Solve the math problems; show reasoning and the final numeric answer.
Example 1:
Problem: <Jenna distance question>
Answer: <worked solution>
Problem: <target question>
Answer: <generation starts here>

The missing answer cue can elicit continuation with another example or problem instead of an answer to the target question. The observed failure mode is illustrated schematically below, rather than quoted from a specific output:

prompt ends:  Problem: <target question>
continuation: Example 2:\nProblem: <another question> ...
scoring:      truncate at the new Example/Problem marker
scored text:  ""
reward:       0  # target question was not answered

Under the same evaluation decoding settings and 256-token output budget, the untrained Qwen3-1.7B scores 41.24% with the authors' prompt and 67.32% with my prompt. Adding only Solution: to the authors' prompt raises the untrained score to 60.50% and eliminates the 168 empty scoring prefixes caused by continuation with a new example or problem. This indicates that prompt-induced output behavior contributes substantially to the low starting accuracy.

Below are my independent three-seed results. Both settings use Qwen3-1.7B, 200 outer steps, group size 8, 30% prompt KV retention, and dense greedy evaluation on all 1,319 GSM8K test questions with a 256-token output limit. All seeds are included. "Corrected" uses the original group-size denominator; "uncorrected" uses the accepted-trajectory denominator.

My prompt setting

MethodSeed 0Seed 1Seed 2Mean ± SD
MaskLift69.29%73.69%72.18%71.72 ± 2.23%
Corrected Sparse-RL76.72%77.18%65.81%73.24 ± 6.44%
Uncorrected Sparse-RL70.51%71.04%58.23%66.59 ± 7.25%

The authors' prompt setting

MethodSeed 0Seed 1Seed 2Mean ± SD
MaskLift76.57%69.29%65.05%70.31 ± 5.83%
Corrected Sparse-RL59.44%63.23%59.51%60.73 ± 2.17%
Uncorrected Sparse-RL59.51%67.48%60.42%62.47 ± 4.36%

With three seeds, the means and sample standard deviations are descriptive.

Corrected Sparse-RL outperforms uncorrected Sparse-RL in my prompt setting by 6.1–7.6 percentage points across the three seeds. Interestingly, it performs worse in the authors' prompt setting, with a mean decrease of 1.74 percentage points. MaskLift is 1.52 percentage points below corrected Sparse-RL on average in my setting, losing on two seeds and winning on one, while exceeding it by 9.58 percentage points on average in the authors' setting.

The authors' reported relative advantage is successfully reproduced under their prompt setting, and their method shows a real advantage there, although the exact scores and gap are not fully reproduced. However, their method does not show an average advantage over corrected Sparse-RL under a more reasonable prompt setting that does not encourage the model to continue with another question instead of answering the target question. The scope and robustness of the advantage therefore remain insufficiently established. Furthermore, in the authors' matched 200-loop, five-seed comparison on one H200, Dense GRPO achieves 74.24% test accuracy in 0.844 hours per run, whereas MaskLift achieves 68.43% in 1.188 hours. Thus, MaskLift achieves lower accuracy with greater end-to-end training cost in this setting. Together, these concerns support maintaining my original score. A future direction is to make the objective more robust. Finally, I really appreciate the authors' willingness and courage to open-source their code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.