MaskLift: Exact Mask Replay for RL under KV-Cache Compression

Online RL post-training spends most of its memory budget on the rollout phase, where the Key-Value (KV) cache of long prompts is the part of the footprint that grows with context and batch. Compressing that cache relieves the pressure but breaks the assumption the policy gradient rests on: trajectories are sampled under a compressed context while the learner scores them under the full one, so the update is off-policy by construction. Remedies for it correct the resulting estimator; we remove what it estimates. MaskLift records the discrete retention mask the rollout compressor produced for each sequence and replays it in the learner's attention, so the sparse/dense importance ratio is identically 1 rather than reweighted or rejected. That identity is exact at the parameters that produced the trajectory, and it is a property of the sparse policy; the dense policy deployment uses inherits the improvement only through a second term, which detaches the RL-improved sparse pass and fits the dense pass to it. On GSM8K under our retention rule, where every uncorrected-learner run finishes below the untrained checkpoint's 40.7, MaskLift reaches 72.8 over its surviving runs against 72.3 for a dense-cache learner that never compressed anything, collapsing on 1 of 5 runs against 0 of 5. It tracks that learner on three further datasets, and we map where the mismatch is severe and where it costs nothing. Qwen3-1.7B, contexts up to 4K; the memory saving is confined to the rollout-decode phase.