- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes LongStraw, a memory-efficient framework for long-context RL policy training. The main idea is to capture the long prompt's KV cache and recurrent states without autograd, then perform block-wise backward replay over responses and a short differentiable prompt tail. The prefix is recaptured after each optimizer update to maintain consistency with the updated model. Experiments demonstrate million-token policy updates on a single H200 and learning gains on long-context QA tasks.
Critical strengths
Memory-efficient RL training on million-token contexts is an important systems and training problem.
Critical weaknesses
1. Weak baseline comparisons. In Section 5.2, the native baseline uses Hugging Face KV-cache storage, while LongStraw uses optimized compact KV-cache pages. The baseline OOMs during prefix capture, before backward even starts. This makes the comparison less convincing, as it's unclear whether the improvement comes from better KV-cache management or the proposed backward method. It would be beneficial to compare against a baseline with the same KV-cache optimization.
2. Unconvincing learning evaluation. The evaluation mostly relies on synthetic SQuAD QA, where the question is always at the end of the prompt and the response is capped at 32 tokens for the 1M experiment. This is a fairly limited workload for long-context RL training. Comparing Tail32 against fully detached training only shows the benefit of adding a short differentiable tail. It doesn't demonstrate why training with a complete million-token prefix is better than using shorter contexts.
3. Limited technical novelty. Most of the techniques, such as prefix caching, tensor memory management, gradient truncation, and activation recomputation, have already been studied. The proposed compact KV-cache storage also seems more like an implementation optimization rather than a new algorithm. It's unclear what the main technical innovation is beyond combining these existing techniques.
Questions for authors
See weaknesses
3 comments
Response to Reviewer qvW5 (2/2)
- Comparison with full-prompt training and 32K retrieval on
Multi-LexSum(Section 5.4 and Appendix L.6, Table 25). Full-prompt backward does not fit at 1M on one H200 (Section 4.1). We therefore used civil-rights case files at 64K–128K tokens, near its memory limit with layer checkpointing and CPU offload, alongside a 32K retrieval-selected baseline (171 test cases, three seeds, 64 updates, 944–1,059-token summaries; gains computed before rounding):
| Method (training and evaluation input) | ROUGE-L initial | Final | Gain | Gain per seed |
|---|---|---|---|---|
LongStraw Tail32 (complete 64K–128K) | 24.65 | 25.04 | +0.40 | +0.69, +0.17, +0.33 |
| Native full-prompt LoRA-RL (complete 64K–128K) | 24.67 | 24.87 | +0.19 | +0.19, +0.09, +0.30 |
| Short-context LoRA-RL (32K retrieval-selected) | 24.43 | 24.63 | +0.20 | +0.21, +0.30, +0.10 |
Training and evaluating on the complete case files reaches a higher final score than 32K retrieval (25.04 vs. 24.63), and Tail32 gains more ROUGE-L than native full-prompt training in all three seeds without its backward memory.
- Distant-context use and question placement at 1M (Section 5.3, Table 15). With supporting passages 105K–944K tokens before the end, the trained policy scores 78.07% EM on the complete 1M input vs. 19.33% on the question alone and 0.00% on the last 8K tokens. Because LongStraw sets by token index and retains the exact state of all earlier tokens, the question need not be at the end.
- Tail-length sweep at 128K across 13 paired seeds (Section 5.1, Appendix L.3, Table 21):
| Gradient scope | EM gain (points) | Seed 95% interval | Peak GiB | GPU-hours |
|---|---|---|---|---|
| 0.00 | no change in any seed | 53.00 | 3.30 | |
| +0.16 | [−0.41, 0.73] | 53.01 | 3.42 | |
| +0.96 | [−0.02, 1.94] | 53.02 | 3.46 | |
| +1.44 | [0.25, 2.63] | 53.01 | 3.57 | |
(Tail32) | +1.12 | [0.14, 2.10] | 53.02 | 3.61 |
| +1.12 | [0.11, 2.13] | 53.02 | 3.66 | |
| Full prompt | +1.52 | [0.33, 2.72] | 67.55 | 5.69 |
Weakness 3: limited technical novelty
KV paging, checkpointing and gradient truncation each exist; our contribution is the unified training-state structure and capture–replay–recapture lifecycle for grouped RL, and as Tables 19–20 in Weakness 1 show, neither native autograd with the same compact cache, nor Prefix Grouper, nor OOMB completed an exact-attention 1M update in this setting.
- Versioned shared-prefix / private-branch state (, Sections 3.2 and 4.1): compact per-layer K/V pages and recurrent/convolution states record ownership, positions, gradient boundary and version , sharing one prefix across responses while isolating rollout and replay buffers.
- Capture–replay–recapture (Section 4.2, Algorithm 1): autograd-free capture, blockwise reverse replay with live memory independent of and , and recapture after each step (stale state shifts log-probabilities by 0.0137 to 3.774 over 1–8 updates; Section 5.2).
- First-token gradient boundary (Section 3.2, Proposition 1): under a frozen readout, removes the first answer token's direct adapter gradient; restores it.
We hope these results address your concerns, and we would welcome further discussion.
Brief Follow-up Before the Discussion Deadline
Dear Reviewer qvW5,
With the discussion deadline less than 24 hours away, we wanted to follow up briefly on our rebuttal and manuscript updates. We added the same-cache native autograd baseline on Qwen3.6-27B (which still OOMs in backward at 1M, whereas LongStraw cuts 512K peak memory by 45.1% and completes 1M updates in 98.95 GiB) alongside 1M comparisons against Prefix Grouper and OOMB. To address your evaluation points, we expanded beyond 32-token SQuAD to three-seed runs on LongMemEval_M (+5.56 EM / +8.40 F1), Multi-LexSum (where complete-context Tail32 outperforms both 32K retrieval at 25.04 vs. 24.63 ROUGE-L and full-prompt training), 4M inputs with 512-token outputs (+4.51 EM / +3.74 F1), and a 1M distant-context control (78.07% vs. 0.00% on the last 8K tokens), while clarifying how our versioned state structure and capture-replay-recapture lifecycle work jointly.
Thank you very much for your service and time in reviewing our work. We would welcome any final feedback or questions before the forum closes.
Best, The Authors
Response to Reviewer qvW5 (1/2)
Dear Reviewer qvW5,
Thank you for your review. We ran the same-cache baseline you suggested, added the new experiments to the revised manuscript (marked in blue), and address each of your three concerns in order below.
Weakness 1: "compare against a baseline with the same KV-cache optimization"
We ran this baseline (Section 5.2 and Appendix L.1, Table 19). Given LongStraw's compact K/V pages and recurrent state, native autograd no longer fails during prefix capture, but it still runs out of memory in backward at 1M, while LongStraw completes the update. Both components are therefore required at 1M: compact storage makes prefix capture fit, and reverse replay makes backward fit.
Qwen3.6-27B(NF4, rank-16 LoRA,Tail32, G=4, R=512, identical inputs and initial adapters; peak allocated GiB / median update seconds over three warm runs; Table 19):Prefix GrouperandOOMBonQwen2.5-7B-Instruct-1M(rank-16 LoRA, G=4, R=512, one H200; update seconds / peak GiB; Appendix L.2, Table 20):Prefix Grouper(full prompt)Prefix Grouper+ CPU offloadOOMBdense (full prompt, exact attention)OOMBSparse64 (full prompt, 64-page sparse)Tail32, exact attention)In
OOMBdense mode, each 4,096-token chunk attends to and back-propagates into all preceding prefix pages (O(P2) work) and repeats the prompt pass for each of the four responses (7.9× slower than LongStraw at 128K, and no update completed after 28.7 hours at 1M when stopped).Sparse64finishes by attending to only 64 selected pages (8,192 tokens) per chunk, approximating attention. LongStraw keeps exact attention over the whole prefix with O(P) backward work by capturing the prefix once without autograd, sharing it across the four responses and differentiating only the responses and the 32-token tail.Weakness 2: "unconvincing learning evaluation"
Beyond 1M QA, the evaluation now covers three-seed 2M–4M QA with 512-token outputs, 96 independent million-token chat histories on
LongMemEval_M, a head-to-head comparison with full-prompt training and 32K retrieval onMulti-LexSum, and a 13-seed tail-length sweep at 128K.LongMemEval_M(Section 5.4, Table 2; Appendix L.5, Table 24). Each question has its own multi-session history of 1.01–1.15 million tokens (about 500 sessions; evidence spans 0.004%–99.7% of the history). Across three seeds on 96 independent held-out histories, EM and F1 rise in every seed: