acceptodds
Under review as a conference paper at ICLR 2027

LongStraw: Memory-Constrained Policy Updates with Million-Token Prefixes

Abstract

Policy training on long histories remains memory intensive even with low-rank adapters, because sequence state and activations grow with the input. We introduce LongStraw, a framework for memory-constrained policy updates on complete long inputs. It couples a compact, versioned prefix-state data structure with a capture–replay–recapture algorithm. The structure organizes layer-specific continuation tensors with their ownership, positions and gradient boundary, sharing prompt state across responses while keeping their extensions private. Capture constructs this state without autograd; block-wise reverse replay differentiates responses and an optional short prompt tail; recapture refreshes the state after each optimizer step. This bounds live activation memory while preserving access to the complete input. The update optimizes a conditional objective that omits gradients through construction of the captured prefix. On one H200, LongStraw completes million-token Qwen3.6-27B updates where native CPU-offloaded autograd of the same objective runs out of memory. On a fixed million-token question-answering benchmark, exact-match accuracy improves by 2.03 percentage points across paired training seeds. Experiments show learning gains on complete inputs up to 4M tokens, and distributed GLM-5.2 experiments demonstrate online policy updates. These results show how jointly organizing prefix state, backward computation and state refresh makes policy training on complete long histories feasible under memory constraints.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
long-context reinforcement learning, memory-efficient training, group relative policy optimization, prompt gradient truncation, activation checkpointing, prefix reuse, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_qvW5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes LongStraw, a memory-efficient framework for long-context RL policy training. The main idea is to capture the long prompt's KV cache and recurrent states without autograd, then perform block-wise backward replay over responses and a short differentiable prompt tail. The prefix is recaptured after each optimizer update to maintain consistency with the updated model. Experiments demonstrate million-token policy updates on a single H200 and learning gains on long-context QA tasks.

Critical strengths

Memory-efficient RL training on million-token contexts is an important systems and training problem.

Critical weaknesses

1. Weak baseline comparisons. In Section 5.2, the native baseline uses Hugging Face KV-cache storage, while LongStraw uses optimized compact KV-cache pages. The baseline OOMs during prefix capture, before backward even starts. This makes the comparison less convincing, as it's unclear whether the improvement comes from better KV-cache management or the proposed backward method. It would be beneficial to compare against a baseline with the same KV-cache optimization.

2. Unconvincing learning evaluation. The evaluation mostly relies on synthetic SQuAD QA, where the question is always at the end of the prompt and the response is capped at 32 tokens for the 1M experiment. This is a fairly limited workload for long-context RL training. Comparing Tail32 against fully detached training only shows the benefit of adding a short differentiable tail. It doesn't demonstrate why training with a complete million-token prefix is better than using shorter contexts.

3. Limited technical novelty. Most of the techniques, such as prefix caching, tensor memory management, gradient truncation, and activation recomputation, have already been studied. The proposed compact KV-cache storage also seems more like an implementation optimization rather than a new algorithm. It's unclear what the main technical innovation is beyond combining these existing techniques.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer qvW5 (1/2)

Dear Reviewer qvW5,

Thank you for your review. We ran the same-cache baseline you suggested, added the new experiments to the revised manuscript (marked in blue), and address each of your three concerns in order below.

Weakness 1: "compare against a baseline with the same KV-cache optimization"

We ran this baseline (Section 5.2 and Appendix L.1, Table 19). Given LongStraw's compact K/V pages and recurrent state, native autograd no longer fails during prefix capture, but it still runs out of memory in backward at 1M, while LongStraw completes the update. Both components are therefore required at 1M: compact storage makes prefix capture fit, and reverse replay makes backward fit.

  1. Same compact cache on Qwen3.6-27B (NF4, rank-16 LoRA, Tail32, G=4G = 4, R=512R = 512, identical inputs and initial adapters; peak allocated GiB / median update seconds over three warm runs; Table 19):
InputNative, same compact cacheNative + CPU offload of saved tensorsLongStraw (ours)
128K52.82 / 43.8948.81 / 62.5138.03 / 58.82
512K114.72 / 298.79111.45 / 319.7062.94 (−45.1%) / 352.57
1MOOM in backwardOOM in backward98.95 / 1068.32
  1. Head-to-head with Prefix Grouper and OOMB on Qwen2.5-7B-Instruct-1M (rank-16 LoRA, G=4G = 4, R=512R = 512, one H200; update seconds / peak GiB; Appendix L.2, Table 20):
System (gradient scope)128K1M
Prefix Grouper (full prompt)67.18 / 94.95OOM
Prefix Grouper + CPU offload68.03 / 71.99OOM
OOMB dense (full prompt, exact attention)1349.66 / 19.16Stopped at 28.7 h (0 steps)
OOMB Sparse64 (full prompt, 64-page sparse)463.21 / 17.63, cold5830.04 / 18.05
LongStraw (Tail32, exact attention)171.45 / 22.845255.38 / 69.25

In OOMB dense mode, each 4,096-token chunk attends to and back-propagates into all preceding prefix pages (O(P2)O(P^2) work) and repeats the prompt pass for each of the four responses (7.9× slower than LongStraw at 128K, and no update completed after 28.7 hours at 1M when stopped). Sparse64 finishes by attending to only 64 selected pages (8,192 tokens) per chunk, approximating attention. LongStraw keeps exact attention over the whole prefix with O(P)O(P) backward work by capturing the prefix once without autograd, sharing it across the four responses and differentiating only the responses and the 32-token tail.

Weakness 2: "unconvincing learning evaluation"

Beyond 1M QA, the evaluation now covers three-seed 2M–4M QA with 512-token outputs, 96 independent million-token chat histories on LongMemEval_M, a head-to-head comparison with full-prompt training and 32K retrieval on Multi-LexSum, and a 13-seed tail-length sweep at 128K.

  1. Longer outputs and three seeds at 2M/4M (Section 5.4, Table 2; Appendix L.4, Table 23). The 32-token cap is used only at 1M; 2M and 4M use a 512-token budget with natural EOS, and F1 rises in all six runs (percentage points):
Input (H200 GPUs)Seed 4101: EM / F1Seed 4102: EM / F1Seed 4103: EM / F13-seed mean: EM / F1
2M (4)+1.04 / +2.27−1.04 / +0.69+1.04 / +1.42+0.35 / +1.46
4M (8)+6.25 / +4.88+6.25 / +5.65+1.04 / +0.70+4.51 / +3.74
  1. Natural million-token chat histories on LongMemEval_M (Section 5.4, Table 2; Appendix L.5, Table 24). Each question has its own multi-session history of 1.01–1.15 million tokens (about 500 sessions; evidence spans 0.004%–99.7% of the history). Across three seeds on 96 independent held-out histories, EM and F1 rise in every seed:
SeedEM (%)ΔEMF1 (%)ΔF1
410122.92 → 27.08+4.1732.28 → 40.71+8.43
410222.92 → 29.17+6.2532.28 → 42.44+10.17
410322.92 → 29.17+6.2532.28 → 38.89+6.61
3-seed mean22.92 → 28.47+5.5632.28 → 40.68+8.40
AuthorsOpenReview ↗

Response to Reviewer qvW5 (2/2)

  1. Comparison with full-prompt training and 32K retrieval on Multi-LexSum (Section 5.4 and Appendix L.6, Table 25). Full-prompt backward does not fit at 1M on one H200 (Section 4.1). We therefore used civil-rights case files at 64K–128K tokens, near its memory limit with layer checkpointing and CPU offload, alongside a 32K retrieval-selected baseline (171 test cases, three seeds, 64 updates, 944–1,059-token summaries; gains computed before rounding):
Method (training and evaluation input)ROUGE-L initialFinalGainGain per seed
LongStraw Tail32 (complete 64K–128K)24.6525.04+0.40+0.69, +0.17, +0.33
Native full-prompt LoRA-RL (complete 64K–128K)24.6724.87+0.19+0.19, +0.09, +0.30
Short-context LoRA-RL (32K retrieval-selected)24.4324.63+0.20+0.21, +0.30, +0.10

Training and evaluating on the complete case files reaches a higher final score than 32K retrieval (25.04 vs. 24.63), and Tail32 gains more ROUGE-L than native full-prompt training in all three seeds without its backward memory.

  1. Distant-context use and question placement at 1M (Section 5.3, Table 15). With supporting passages 105K–944K tokens before the end, the trained policy scores 78.07% EM on the complete 1M input vs. 19.33% on the question alone and 0.00% on the last 8K tokens. Because LongStraw sets P−bP-b by token index and retains the exact state of all earlier tokens, the question need not be at the end.
  2. Tail-length sweep at 128K across 13 paired seeds (Section 5.1, Appendix L.3, Table 21):
Gradient scopeEM gain (points)Seed 95% intervalPeak GiBGPU-hours
b=0b = 00.00no change in any seed53.003.30
b=1b = 1+0.16[−0.41, 0.73]53.013.42
b=4b = 4+0.96[−0.02, 1.94]53.023.46
b=8b = 8+1.44[0.25, 2.63]53.013.57
b=32b = 32 (Tail32)+1.12[0.14, 2.10]53.023.61
b=128b = 128+1.12[0.11, 2.13]53.023.66
Full prompt+1.52[0.33, 2.72]67.555.69

Weakness 3: limited technical novelty

KV paging, checkpointing and gradient truncation each exist; our contribution is the unified training-state structure and capture–replay–recapture lifecycle for grouped RL, and as Tables 19–20 in Weakness 1 show, neither native autograd with the same compact cache, nor Prefix Grouper, nor OOMB completed an exact-attention 1M update in this setting.

  1. Versioned shared-prefix / private-branch state (SPkS_P^k, Sections 3.2 and 4.1): compact per-layer K/V pages and recurrent/convolution states record ownership, positions, gradient boundary and version kk, sharing one prefix across GG responses while isolating rollout and replay buffers.
  2. Capture–replay–recapture (Section 4.2, Algorithm 1): autograd-free capture, blockwise reverse replay with live memory independent of PP and GG, and recapture after each step (stale state shifts log-probabilities by 0.0137 to 3.774 over 1–8 updates; Section 5.2).
  3. First-token gradient boundary (Section 3.2, Proposition 1): under a frozen readout, b=0b=0 removes the first answer token's direct adapter gradient; b≥1b \ge 1 restores it.

We hope these results address your concerns, and we would welcome further discussion.

AuthorsOpenReview ↗

Brief Follow-up Before the Discussion Deadline

Dear Reviewer qvW5,

With the discussion deadline less than 24 hours away, we wanted to follow up briefly on our rebuttal and manuscript updates. We added the same-cache native autograd baseline on Qwen3.6-27B (which still OOMs in backward at 1M, whereas LongStraw cuts 512K peak memory by 45.1% and completes 1M updates in 98.95 GiB) alongside 1M comparisons against Prefix Grouper and OOMB. To address your evaluation points, we expanded beyond 32-token SQuAD to three-seed runs on LongMemEval_M (+5.56 EM / +8.40 F1), Multi-LexSum (where complete-context Tail32 outperforms both 32K retrieval at 25.04 vs. 24.63 ROUGE-L and full-prompt training), 4M inputs with 512-token outputs (+4.51 EM / +3.74 F1), and a 1M distant-context control (78.07% vs. 0.00% on the last 8K tokens), while clarifying how our versioned state structure and capture-replay-recapture lifecycle work jointly.

Thank you very much for your service and time in reviewing our work. We would welcome any final feedback or questions before the forum closes.

Best, The Authors

Reviewer_ykyGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LongStraw addresses memory-constrained policy optimization with extremely long prompts. It combines compact, versioned prefix states with a capture–replay–recapture procedure: the prefix is processed without autograd, responses and an optional short prompt tail are differentiated through blockwise replay, and prefix states are refreshed after parameter updates. The method retains access to the complete input while omitting gradients through the earlier prefix computation. Experiments demonstrate million-token updates on a single H200, learning gains on constructed long-context QA tasks, and distributed update feasibility on GLM-5.2.

Critical strengths
  1. The paper addresses a practically important training bottleneck. The implementation demonstrates substantial engineering effort in managing prefix-state ownership, activation lifetimes, and replay. Comparisons against native autograd under the same conditional objective provide useful evidence of improved memory efficiency.

  2. The first-answer-token analysis provides a clear and actionable insight. Under a frozen readout, detaching the entire prompt removes the first answer token’s direct gradient to decoder adapters. The theoretical analysis and matched gradient-scope experiments help explain why retaining a short differentiable tail improves over complete detachment.

Critical weaknesses
  1. The justification for the gradient boundary remains limited. Proposition 1 establishes that retaining at least one prompt token restores a direct gradient path, but does not establish why 32 tokens are sufficient or when omitting earlier prefix gradients should have little effect. The strongest comparisons against full-prompt training concentrate on Qwen3.6-27B and constructed SQuAD-based QA tasks. The 1M evaluation contains only 96 questions sharing four archives, and the confidence intervals cross zero when article/archive variability is included. Although the additional RULER and GLM experiments broaden the evaluation, they do not establish that Tail32 reliably preserves full-prompt learning across models and tasks.

  2. The incremental systems contribution needs stronger direct comparison with existing approaches. Prefix Grouper already shares prefix computation while retaining differentiability, and OOMB combines chunkwise recomputation, paged K/V and gradient storage, and CPU offloading. The paper includes useful component ablations, but the main resource comparisons do not sufficiently isolate the benefit of the proposed organization and replay from existing techniques or improvements to native cache storage. This is particularly relevant because the native 1M baseline fails during gradient-free capture, before response backward.

Questions for authors
  1. Could you provide a systematic tail-length ablation, including b=1,4,8,32b=1,4,8,32 and longer tails, with matched data and update counts? Please report both learning performance and resource costs. Is 32 primarily an implementation choice, or does it offer a reproducible quality–memory advantage?

  2. At lengths where full-prompt training is feasible, could you quantify the magnitude and alignment of the omitted gradient contribution? Could the learning comparison also cover longer answers and more independent evaluation archives, with a predefined non-inferiority margin if performance preservation is the intended claim?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ykyG (1/2)

Dear Reviewer ykyG,

Thank you for your thoughtful review and for highlighting our first-answer-token analysis. We have added the experiments you requested to the revised manuscript (marked in blue) and address each of your points in order below.

Weakness 1 and Question 1: "a systematic tail-length ablation, including b = 1, 4, 8, 32 and longer tails"

From b=8b = 8 tokens on, the differentiable prompt tail recovers most of the full-prompt learning gain (+1.12 to +1.44 vs. +1.52 EM points) at 53.0 GiB peak memory instead of 67.55 GiB (Section 5.1 and Appendix L.3, Table 21). Here bb is the number of final prompt tokens receiving gradients; all earlier prompt tokens remain attended to through the captured prefix state.

We trained all seven gradient scopes at 128K on Qwen3.6-27B with the same data, 12 parameter-changing updates, 13 paired seeds per scope and 96 held-out questions, using native autograd on every row to isolate bb from our replay kernel (GPU-hours cover sampling, updates and evaluation):

Tail bbEM gain (points)Seed 95% intervalMean peak GiBMean GPU-hours
00.00no change in any seed53.003.30
1+0.16[−0.41, 0.73]53.013.42
4+0.96[−0.02, 1.94]53.023.46
8+1.44[0.25, 2.63]53.013.57
32 (Tail32)+1.12[0.14, 2.10]53.023.61
128+1.12[0.11, 2.13]53.023.66
Full prompt+1.52[0.33, 2.72]67.555.69

At b=0b = 0, EM is unchanged in all 13 seeds, matching Proposition 1. Thus b=32b = 32 is an implementation choice (one 32-token replay block) rather than a tuned optimum; b=8b = 8 gives a slightly higher mean gain.

Weakness 2: systems benefit relative to Prefix Grouper, OOMB and improved native caching

With LongStraw's compact cache, native autograd still runs out of memory in backward at 1M (Table 19); on Qwen2.5-7B-Instruct-1M at 1M, Prefix Grouper runs out of memory and OOMB dense had not completed an update after 28.7 hours when stopped, whereas LongStraw finishes in 1.5 hours (Section 5.2, Appendix L.1–L.2, Table 20).

  1. Same compact state on Qwen3.6-27B (one H200, Tail32, G=4G=4, R=512R=512; peak GiB / median seconds; Table 19):
InputNative, same compact stateNative + CPU offload of saved tensorsLongStraw (ours)
128K52.82 / 43.8948.81 / 62.5138.03 / 58.82
512K114.72 / 298.79111.45 / 319.7062.94 (−45.1%) / 352.57
1MOOM in backwardOOM in backward98.95 / 1068.32
  1. Head-to-head with Prefix Grouper and OOMB on Qwen2.5-7B-Instruct-1M (one H200, rank-16 LoRA, G=4G=4, R=512R=512; update seconds / peak GiB; Table 20):
System (gradient scope)128K1M
Prefix Grouper (full prompt)67.18 / 94.95OOM
Prefix Grouper + CPU offload68.03 / 71.99OOM
OOMB dense (full prompt, exact attention)1349.66 / 19.16Stopped at 28.7 h (0 steps)
OOMB Sparse64 (full prompt, 64-page sparse)463.21 / 17.63, cold5830.04 / 18.05
LongStraw (Tail32, exact attention)171.45 / 22.845255.38 / 69.25

OOMB dense attends to and back-propagates through all preceding prefix pages for each 4,096-token chunk (O(P2)O(P^2) work) and repeats the prompt pass G=4G=4 times (7.9× slower than LongStraw at 128K). Sparse64 finishes at 1M by retrieving only 64 pages (8,192 tokens) per chunk, approximating attention. LongStraw preserves exact full-prefix attention with O(P)O(P) backward work by capturing the prefix once without autograd, sharing it across responses and differentiating only the responses and the 32-token tail.

AuthorsOpenReview ↗

Response to Reviewer ykyG (2/2)

Question 2(a): magnitude and alignment of the omitted gradient

Tail32 and Tail128 gradients nearly coincide, whereas the earlier prefix contributes a substantial part of the full-prompt gradient; Table 21 in (1/2) above measures what omitting it costs in end-to-end learning (Appendix L.3, Table 22). Over all 992 LoRA tensors for fixed responses (at initialization and after one update):

Scope (reference, input)CasesRelative L2 differenceCosine
b=0b = 0 (vs. Tail128, 128K)3 seeds × 2 states0.892–1.0000.007–0.454
b=1b = 1 (vs. Tail128, 128K)3 seeds × 2 states0.791–0.8740.490–0.613
b=4b = 4 (vs. Tail128, 128K)3 seeds × 2 states0.355–0.5400.843–0.935
b=8b = 8 (vs. Tail128, 128K)3 seeds × 2 states0.243–0.3030.953–0.970
b=32b = 32 (vs. Tail128, 128K)3 seeds × 2 states0.021–0.0360.9994–0.9998
Tail32 (vs. full prompt, 32K)1 seed × 2 states0.526, 0.5330.846, 0.851
Full prompt repeated (32K)1 seed × 2 states0.017, 0.0180.9998

In task learning, omitting the earlier-prefix gradient changes mean EM gain at 128K by 0.40 points (+1.12 vs. +1.52) while saving 14.53 GiB of peak memory and 36% of GPU time, and unlocks 1M–4M training where full-prompt backward runs out of memory.

Question 2(b): longer answers, more independent evaluation sources and comparison with full-prompt training

We added three-seed 2M/4M QA with 512-token outputs, LongMemEval_M with 96 independent million-token histories, and a direct comparison with full-prompt LoRA-RL on 1,000-token Multi-LexSum legal summaries (Section 5.4, Appendix L.4–L.6).

  1. Longer answers at 2M and 4M (16 updates, 512-token cap with natural EOS, three seeds; Table 23). F1 rises in all six runs and EM in five (percentage points):
Input (H200 GPUs)Seed 4101: EM / F1Seed 4102: EM / F1Seed 4103: EM / F13-seed mean: EM / F1
2M (4)+1.04 / +2.27−1.04 / +0.69+1.04 / +1.42+0.35 / +1.46
4M (8)+6.25 / +4.88+6.25 / +5.65+1.04 / +0.70+4.51 / +3.74
  1. 96 independent evaluation histories on LongMemEval_M (1.01–1.15M tokens per history; Table 24). EM and F1 rise in every seed (95% question-resampling interval of mean ΔF1: [3.48, 13.90]):
SeedEM (%)ΔEMF1 (%)ΔF1
410122.92 → 27.08+4.1732.28 → 40.71+8.43
410222.92 → 29.17+6.2532.28 → 42.44+10.17
410322.92 → 29.17+6.2532.28 → 38.89+6.61
3-seed mean22.92 → 28.47+5.5632.28 → 40.68+8.40
  1. Head-to-head on Multi-LexSum legal cases at 64K–128K tokens (171 test cases, three seeds, 64 updates, 944–1,059-token summaries, gains computed before rounding; Table 25), near the memory limit of native full-prompt LoRA-RL with layer checkpointing and CPU offload:
Method (input)ROUGE-L initialFinalGainGain per seed
LongStraw Tail32 (complete 64K–128K)24.6525.04+0.40+0.69, +0.17, +0.33
Native full-prompt LoRA-RL (complete 64K–128K)24.6724.87+0.19+0.19, +0.09, +0.30

Rather than setting a post-hoc non-inferiority margin, we report every seed: Tail32 gains more ROUGE-L than native full-prompt training in all three seeds without full-prompt backward memory.

We hope these results address your concerns, and we would welcome further discussion.

AuthorsOpenReview ↗

Brief Follow-up Before the Discussion Deadline

Dear Reviewer ykyG,

Since the discussion period closes in under 24 hours, we wanted to send a brief follow-up on our reply and revised draft. To address your questions, we swept seven tail scopes (from b = 0 to full prompt) across 13 seeds at 128K, showing that b >= 8 recovers nearly all full-prompt gains (+1.12 to +1.44 vs. +1.52 EM) while saving 14.53 GiB and 36% of GPU time, and quantified the omitted-gradient alignment across all 992 LoRA tensors. We also added three-seed evaluations on longer outputs and independent histories (Multi-LexSum, where Tail32 beats native full-prompt LoRA-RL at +0.40 vs. +0.19 ROUGE-L, and LongMemEval_M at +8.40 F1), alongside 1M comparisons against Prefix Grouper, OOMB, and storage-matched native autograd (-45.1% peak memory at 512K).

We truly appreciate the time and thought you put into reviewing our paper. Whenever you have a spare moment, we would be grateful to hear if these updates resolve your concerns.

Best regards, The Authors

Reviewer_EPc9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LongStraw, a framework for policy updates on complete long inputs under GPU memory constraints. It captures a detached prefix state without autograd, shares this state across responses, reconstructs differentiable computations through blockwise reverse replay, and refreshes the state after parameter updates. The method explicitly optimizes a conditional objective that omits gradients through the captured prefix. A short differentiable prompt tail restores the first answer token’s direct adapter gradient when the readout is frozen. Experiments demonstrate million-token updates with Qwen3.6-27B on one H200, learning studies at up to four million tokens, and a distributed GLM-5.2 execution study. Matched shorter-context experiments examine the learning consequences of the gradient boundary.

Critical strengths
  1. The systems contribution addresses a concrete bottleneck and is supported by detailed measurements. Completing a million-token update on one H200 is a meaningful feasibility result. The paper separates retained state from transient workspace, reports host as well as device memory, and includes controlled storage and replay ablations.

  2. The gradient-boundary analysis is useful and experimentally grounded. The first-token result identifies a specific consequence of detaching the entire prompt, rather than treating truncation as an unspecified approximation. The matched 8K study, with eight seeds per arm, shows substantially stronger learning with Tail32 than complete detachment.

Critical weaknesses
  1. The million-token learning evidence is narrower than the headline result suggests. Appendix I states that the +2.03+2.03-point EM improvement does not meet the prespecified 55-point primary criterion. The positive interval [1.09,2.96][1.09,2.96] conditions on the fixed evaluation set; intervals widen to [−3.70,7.93][-3.70,7.93] when articles are resampled and [−7.41,11.52][-7.41,11.52] when archives are also resampled.

  2. There are only 96 evaluation questions sharing four archives, and prominent changes include answer-span formatting. The 2M/4M studies are single-seed experiments, while GLM has no held-out learning evaluation.

  3. The incremental systems benefit over a minimally improved native baseline remains insufficiently isolated. At 1M, Native tail + CPU fails during gradient-free capture, before saved-tensor offloading can help. The comparison therefore combines compact prefix storage with the replay algorithm.

Questions for authors
  1. Interpretation of the learning result: Can the main text explicitly report that the prespecified primary criterion was not met and distinguish fixed-benchmark seed uncertainty from uncertainty over articles and archives?

  2. What evidence supports interpreting the observed gain as improved long-context task performance rather than primarily answer-span or termination adaptation?

  3. Decisive systems comparison: Under the same conditional objective, what happens if native tail autograd receives LongStraw’s compact, shared prefix state but does not use reverse block replay?

  4. Numerical validity and portability: Are the reported long-context numerical differences stable over successive optimizer steps?

  5. At 512K, the gradient relative error is about 0.0580.058, while the first AdamW-update relative error is about 0.250.25. Please also clarify the cause of the GLM output-head log-probability check failure and whether it could affect the previously reported online updates.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EPc9 (1/2)

Dear Reviewer EPc9,

Thank you for your careful review and for recognizing the single-H200 million-token result and the gradient-boundary analysis. We have updated the manuscript (marked in blue) and address each of your points in order below.

Weakness 1 and Question 1: interpreting the 1M learning gain and the uncertainty intervals

The 1M gain of +2.03 EM points is below the 5-point criterion prespecified in the submitted Appendix I, while remaining consistently positive across the 18 paired training seeds ([1.09, 2.96], p = 0.000261); as requested, Section 5.3 now reports the seed, article and archive intervals side by side.

  • 1M SQuAD intervals (18 paired seeds): EM rises from 76.04% to 78.07% (+2.03) on 96 questions from 48 held-out articles in four archives, outperforming full detachment (+0.98 points; Section 5.3, Appendix I). Every leave-one-archive-out mean gain is positive (+0.31 to +4.78 points). Resampling the 48 articles or the four archives widens the 95% interval to [−3.70, 7.93] and [−7.41, 11.52], because only four archives exist.
  • 96 independent 1.01–1.15M histories on LongMemEval_M: To evaluate without shared-archive variance, we added LongMemEval_M (Section 5.4, Table 2; Appendix L.5, Table 24), where each question has its own multi-session chat history, so resampling questions also resamples histories. With the 2M recipe, EM and F1 rise in every seed, and the 95% question-resampling interval of the mean F1 gain is [3.48, 13.90]:
SeedEM (%)ΔEMF1 (%)ΔF1
410122.92 → 27.08+4.1732.28 → 40.71+8.43
410222.92 → 29.17+6.2532.28 → 42.44+10.17
410322.92 → 29.17+6.2532.28 → 38.89+6.61
3-seed mean22.92 → 28.47+5.5632.28 → 40.68+8.40

Weakness 2: single-seed 2M/4M studies, the 96-question set and GLM

We ran two additional seeds (4102 and 4103) at both 2M and 4M (F1 rises in all six runs and EM in five), and added the 171-case Multi-LexSum summarization study (Section 5.4, Table 2; Appendix L.4 and L.6, Tables 23 and 25).

  1. Three seeds at 2M and 4M (16 updates, 128 training and 96 held-out questions, 512-token cap with natural EOS; gains in percentage points):
Input (H200 GPUs)Seed 4101: EM / F1Seed 4102: EM / F1Seed 4103: EM / F13-seed mean: EM / F1
2M (4)+1.04 / +2.27−1.04 / +0.69+1.04 / +1.42+0.35 / +1.46
4M (8)+6.25 / +4.88+6.25 / +5.65+1.04 / +0.70+4.51 / +3.74
  1. 171 legal cases on Multi-LexSum (64K–128K tokens): Chosen near the memory limit of native full-prompt LoRA-RL with CPU offload to allow a direct reference on 944–1,059-token summaries (gains computed before rounding). Tail32 gains more ROUGE-L than native full-prompt training in every seed:
Method (input)ROUGE-L initialFinalGainGain per seed
LongStraw Tail32 (complete 64K–128K)24.6525.04+0.40+0.69, +0.17, +0.33
Native full-prompt LoRA-RL (complete 64K–128K)24.6724.87+0.19+0.19, +0.09, +0.30
Short-context LoRA-RL (32K retrieval-selected)24.4324.63+0.20+0.21, +0.30, +0.10
  1. Role of GLM-5.2: Tests distributed execution on a sparse-attention MoE model (five online updates at 2M on 32 GPUs; Section 5.5); held-out learning is evaluated on Qwen3.6-27B.

Weakness 3 and Question 3: "native tail autograd receives LongStraw's compact, shared prefix state but does not use reverse block replay"

We ran this control (Section 5.2 and Appendix L.1, Table 19). With LongStraw's compact prefix pages and recurrent state, native autograd completes prefix capture at 1M, but both native variants still run out of memory in backward, while LongStraw completes the update.

All arms use Qwen3.6-27B (NF4), rank-16 LoRA, Tail32, G=4G = 4, R=512R = 512, per-layer checkpointing, and identical inputs and initial adapters (peak allocated GiB / median seconds over three warm runs):

InputNative, same compact stateNative + CPU offload of saved tensorsLongStraw (ours)
128K52.82 / 43.8948.81 / 62.5138.03 / 58.82
512K114.72 / 298.79111.45 / 319.7062.94 (−45.1%) / 352.57
1MOOM in backwardOOM in backward98.95 / 1068.32

With prefix storage equalized, blockwise reverse replay cuts the 512K peak by 45.1% and enables the 1M update on one H200; where native backward fits, native is faster. PyTorch's expandable-segment allocator does not change the 1M outcome.

AuthorsOpenReview ↗

Response to Reviewer EPc9 (2/2)

Question 2: long-context task performance versus answer-span or termination adaptation

Shorter answer spans contribute to EM (at 1M, the most common change is "eight languages" → "eight"). Three results show that the improvement also depends on the long context and is not explained by termination:

  1. Information-ablation controls (Appendix Table 15): At 1M, trained EM is 78.07% (+2.03) on the complete input, 19.33% (−1.50) on the question alone and 0.00% on the last 8K tokens, with supporting passages 105K–944K tokens before the end.
  2. Natural EOS and first-token probability (Appendix L.4): At 4M, six of the seven newly exact answers already ended with natural EOS before training, scoring strips punctuation, case and articles, and gold first-token probability rises from 73.28% → 80.68% (1M), 57.58% → 60.09% (2M) and 24.66% → 37.36% (4M).
  3. 1,000-token free-form summaries (Table 25 above): On Multi-LexSum, ROUGE-L rises in all three seeds, where there is no short answer span to trim.

Questions 4 and 5: successive optimizer steps, the 512K AdamW error and the GLM output-head check

Over three successive optimizer steps, the AdamW update difference falls from 0.183 to 0.061 at 128K and from 0.251 to 0.068 at 512K, and the GLM-5.2 check failed only because of its FP32 CPU reference (Sections 5.2 and 5.5; Appendix L.7, Tables 26–27).

  1. Successive optimizer steps on Qwen3.6-27B (Table 26): At each of three successive states along the native trajectory, native autograd and LongStraw start from identical parameters, Adam moments and step counters, recapture the prefix and score fixed responses across all 992 LoRA tensors:
InputStepGradient relative L2Gradient cosineAdamW update relative L2Max log-prob diff
128K10.0240.99970.1830.069
128K20.0240.99970.0830.114
128K30.0320.99950.0610.246
512K10.0580.99840.2510.074
512K20.0840.99660.1520.097
512K30.0370.99930.0680.116

At step 1, m^1=g1\hat{m}_1 = g_1 and v^1=g12\hat{v}_1 = g_1^2, so each coordinate moves by roughly η,sign⁡(g1)\eta\\,\operatorname{sign}(g_1) regardless of magnitude; near-zero coordinates that flip sign therefore create a 0.251 update difference from a 0.058 gradient difference. Once v^t\hat{v}_t accumulates across steps, this sign sensitivity fades and the update difference drops to 0.152 and 0.068.

  1. GLM-5.2 output-head check (tolerance 2e-5, 224 records across 7 updates and 32 ranks; Table 27):
Updates (groups)Max diff vs. FP32 referenceMax diff vs. FP64 referenceOriginal check
1–6 (groups 1, 2, 7, 8, 10, 11)6.91e-6 to 1.31e-54.59e-8 to 1.54e-7pass
7 (group 12)3.72e-59.49e-8fail on all 32 ranks

The stored log-probabilities agree with the FP64 reference within 1.54e-7 at every update; the five reported updates passed the original check and are unaffected.

We hope these results address your concerns, and we would welcome further discussion.

AuthorsOpenReview ↗

Follow-up to Reviewer EPc9

Dear Reviewer EPc9,

Thank you again for your encouraging review and for the thoughtful suggestions, which helped us strengthen the paper substantially. With less than 24 hours left in the discussion period, we wanted to share a brief follow-up on our response and revised manuscript. To address your points, we added the side-by-side seed, article, and archive intervals in Section 5.3, ran the storage-matched native autograd control (confirming that reverse block replay cuts 512K peak memory by 45.1% and prevents 1M backward OOM), expanded our evaluation to three seeds on LongMemEval_M (+5.56 EM / +8.40 F1), Multi-LexSum (+0.40 vs. +0.19 ROUGE-L over full-prompt training), and 2M/4M inputs (+4.51 EM at 4M), and verified multi-step AdamW stability along with the FP64 GLM-5.2 output-head precision (9.49e-8).

We truly appreciate the time and care you volunteered to evaluate our submission. If your schedule permits before the forum closes, we would be grateful for any remaining thoughts or feedback you might have.

Warm regards, The Authors

Reviewer_efCmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

LongStraw attacks the problem of long-context post-training in the million-token regime. In this regime, adapter methods like LoRA and other parameter-efficient-tuning (PEFT) methods still introduce activations sized with the sequence length, leading to OOM issues at long-contexts.

LongStraw attacks this issue by detaching generation from back-propagation. During inference, the model generates the entire response, however backpropagation occurs only over the generated response and a segment of the input prompt (rather than over all the tokens in the combined prompt + generated response). To achieve this, LongStraw proposes a capture-replay-capture algorithm.

Interestingly, this is enough to bring down memory at 1M context length to just under 100GB (figure 4), so effectively solves memory-bottleneck issues.

Critical strengths

The paper clearly shows memory reductions whilst accelerating runtime performance.

Critical weaknesses

Two main criticisms of the paper.

  1. The paper is extremely hard to understand. For example, the messaging in the introduction is unclear, with an unclear attribution to what is novel within this work compared to prior SOTA.
  2. The evaluation is currently contained to two models: Qwen 3.7-27B and GLM-5.2 (that too GLM-5.2 evaluation isn't even over a complete dataset). Moreover, it seems only one dataset is evaluated: SQuad. A more comprehensive model would span multiple models over various datasets.
Questions for authors

Can the authors evaluate across more models and more natively longer-context evaluations to demonstrate the efficicay of the approach?

3 comments
AuthorsOpenReview ↗

Response to Reviewer efCm (1/2)

Dear Reviewer efCm,

Thank you for your review and for noting our memory and runtime gains. We have revised the manuscript (marked in blue), rewriting the Introduction to state the problem, prior-system failure modes and our contributions upfront, and address both of your concerns in order below.

Weakness 1: "unclear attribution to what is novel within this work compared to prior SOTA"

On one H200 at 1M tokens (G=4G=4, R=512R=512), LongStraw completes exact-attention LoRA updates in 18 minutes on Qwen3.6-27B and 1.5 hours on Qwen2.5-7B-Instruct-1M, whereas native autograd with our compact state, Prefix Grouper and OOMB dense all fail to complete an update (Sections 1 and 5.2; Appendix L.1–L.2, Tables 19–20).

System at 1M (one H200, rank-16 LoRA)ModelGradient / attention1M outcome
Native autograd + our compact stateQwen3.6-27BTail32 / exactOOM in backward (512K: 114.72 GiB)
Native + compact state + CPU offloadQwen3.6-27BTail32 / exactOOM in backward (512K: 111.45 GiB)
LongStraw (ours)Qwen3.6-27BTail32 / exact1,068.32 s, 98.95 GiB (512K: 62.94 GiB)
Prefix Grouper (GPU or CPU offload)Qwen2.5-7B-1Mfull prompt / exactOOM
OOMB denseQwen2.5-7B-1Mfull prompt / exactNo update after 28.7 h (stopped)
OOMB Sparse64Qwen2.5-7B-1Mfull prompt / 64 pages5,830.04 s, 18.05 GiB (approximate)
LongStraw (ours)Qwen2.5-7B-1MTail32 / exact5,255.38 s, 69.25 GiB
  • Why prior systems fail at 1M: With our compact state, native autograd finishes 1M capture but still OOMs in backward (−45.1% peak memory with LongStraw at 512K). Prefix Grouper shares the prompt forward pass across responses but differentiates the full prompt and OOMs. OOMB dense differentiates attention over all earlier chunks per chunk (O(P2)O(P^2) work) and repeats the prompt pass per response; Sparse64 finishes only by approximating attention over 64 pages per chunk.
  • Our three contributions:
    1. Versioned shared-prefix state (Section 4.1): stores compact per-layer K/V pages and recurrent/convolution state once across all GG responses, with private per-response extensions.
    2. Capture–replay–recapture (Section 4.2, Algorithm 1): autograd-free prefix capture, blockwise reverse replay over responses and the final b=32b=32 prompt tokens (O(P)O(P) work), and state refresh after each step.
    3. Gradient-boundary analysis (Section 3.2, Proposition 1): under a frozen readout, detaching the prompt (b=0b=0) zeroes the first answer token's direct gradient, and a short tail restores it (0.00 EM gain at b=0b=0 vs. +1.12 to +1.44 at b≥8b \ge 8 and +1.52 for full prompt across 13 seeds at 128K; Table 21).

Weakness 2 / Question: evaluating across more models and natively long-context tasks

Because LongStraw's state abstraction SPk=(ZPk,MPk)S_P^k = (Z_P^k, M_P^k) decouples the update lifecycle from layer internals, it adapts to any LLM by registering its per-layer continuation state: our three evaluated models span all four stateful layer primitives in modern LLMs, and our five benchmarks now include million-token natural chat histories (LongMemEval_M) and multi-document legal summarization (Multi-LexSum).

  1. Adapting to virtually all modern LLM architectures (Sections 4.1, 4.3, 5.1–5.5; Appendix B and L.2): What differs across LLM families for long-context training is the stateful layer primitive that must be captured, branched across GG responses and replayed. Whereas the public releases of Prefix Grouper and OOMB implement only standard full-attention layers, LongStraw supports every stateful layer family in modern open-weight LLMs:
Evaluated modelArchitecture family coveredContinuation state in SPkS_P^kRole in our evaluation
Qwen2.5-7B-1MStandard GQA full attention + dense FFN (Llama / Qwen2.5 style)28-layer paged BF16 K/V cache128K & 1M head-to-head on 1 H200 vs. Prefix Grouper & OOMB (Table 20)
Qwen3.6-27BHybrid linear-recurrent/conv + full attention (Qwen3-Next / Mamba hybrid)16-layer K/V pages + 48-layer Gated DeltaNet state8K–4M on 1–8 H200s; full systems & held-out learning (Tables 1–2, 19, 21–25)
GLM-5.2Compressed sparse attention + routed MoE (DeepSeek-V3 / GLM-5 style)78-layer CP32 MLA latent KV + DSA index keys + EP32 MoE2M on 32 H200s; distributed MoE feasibility & online updates (Section 5.5, Table 27)
AuthorsOpenReview ↗

Response to Reviewer efCm (2/2)

  • Why Qwen3.6-27B updates faster at 1M than Qwen2.5-7B-1M: Only 16 of the 64 Qwen3.6 layers require full-attention KV pages (64 GiB at 1M), while the 48 Gated DeltaNet layers carry a fixed 0.14 GiB recurrent/convolution state at any length (Section 4.1).
    • Why GLM-5.2 runs five online updates at 2M: On the 78-layer MoE GLM-5.2, Megatron OOMs at 2M in all tested TP/CP/EP layouts, and sampling four responses per 2M prompt alone takes a median of 38.2 minutes on 32 H200s (Section 5.5). Its role is thus to validate distributed CP32/EP32 execution and online updates at 2M (with all seven applied updates verified within 1.54e-7 of an FP64 reference; Table 27), while held-out learning is evaluated across five benchmarks on Qwen3.6-27B.
  1. Five long-context benchmarks beyond SQuAD (Sections 5.3–5.4, Appendix L.4–L.6, Tables 2, 16, 23–25):
Benchmark (input length)CapSeedsHeld-out result (per-seed gains in parentheses)
SQuAD long-doc QA (1M)32 tok18EM 76.04% → 78.07% (+2.03); question-only 19.33%, last-8K 0.00% (Table 15)
SQuAD long-doc QA (2M / 4M)512 tok3 each2M ΔF1 +1.46 (+2.27, +0.69, +1.42); 4M ΔEM +4.51, ΔF1 +3.74 (Table 23)
LongMemEval_M chat histories (1.01–1.15M)512 tok3EM 22.92% → 28.47% (+5.56) (+4.17, +6.25, +6.25); F1 32.28% → 40.68% (+8.40) (+8.43, +10.17, +6.61; Table 24)
RULER niah_multivalue (128K)128 tok1recall 89.06% → 99.61% (+10.55) (Table 16)
Multi-LexSum legal cases (64K–128K)2,048 tok3ROUGE-L 24.65 → 25.04 (+0.40) (+0.69, +0.17, +0.33) vs. full prompt +0.19 (+0.19, +0.09, +0.30) and 32K retrieval +0.20 (Table 25)

On LongMemEval_M, each of the 96 held-out questions has its own independent 500-session chat history (1.01–1.15M tokens), and both EM and F1 rise in every seed. On Multi-LexSum (171 civil-rights cases at 64K–128K, near the memory limit of native full-prompt LoRA-RL with CPU offload), Tail32 gains more ROUGE-L on 944–1,059-token summaries than full-prompt training in every seed and reaches the highest final score (25.04 vs. 24.87 and 24.63).

We hope these results address your concerns and welcome further discussion.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal and Additional Experiments

Dear Reviewer efCm,

With less than 24 hours remaining in the discussion period, we wanted to follow up briefly on our rebuttal and revised manuscript. To address your feedback, we rewrote the Introduction to clarify our contributions over prior systems, added 1M comparisons on Qwen2.5-7B-1M (where LongStraw completes exact-attention updates in 69.25 GiB while Prefix Grouper OOMs and OOMB dense stalls past 28 hours), and expanded our evaluation to five benchmarks, including LongMemEval_M (+5.56 EM / +8.40 F1 across three seeds) and Multi-LexSum.

We are very grateful for the time you have volunteered to review our work. If your schedule allows, we would appreciate any thoughts you might have on these updates before the forum closes.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Core Contributions and New Experiments

Dear Area Chair, Senior Area Chair, and Reviewers,

We sincerely thank all four reviewers for their constructive feedback. To assist the Area Chairs and reviewers in navigating our revision, this general response summarizes our core mechanism and the new experiments addressing your primary concerns.

LongStraw rests on a simple observation: conditioning on a million-token history in the forward pass and differentiating through its construction in the backward pass have very different costs and roles. By coupling a compact, versioned prefix-state structure (SPkS_P^k) with capture-replay-recapture, while retaining a short differentiable prompt tail (Tail32) to preserve the first answer token's direct gradient under a frozen readout (Proposition 1), LongStraw bounds live backward memory to O(b+R)O(b + R) while keeping exact attention over the complete input.

Across the reviews, the reservations centered on two fair questions: (1) Systems attribution: Does our 1M capability come from backward replay or merely from compact KV-cache storage, and how does LongStraw compare against Prefix Grouper and OOMB? (2) Learning validity: Does a short differentiable tail preserve full-prompt learning across window sizes and long outputs, and do the gains hold on natural million-token tasks with independent histories and multiple seeds?

We ran every experiment suggested by the reviewers and incorporated them into the revised manuscript (marked in blue in Sections 1 and 5, Table 2, and Appendix L, Tables 19--27):

  1. Isolating backward replay from prefix storage (Section 5.2, Table 19): When native autograd is given LongStraw's exact compact K/V pages and recurrent state on Qwen3.6-27B (G=4G=4, R=512R=512, one H200), it completes 1M prefix capture but still runs out of memory in backward (with or without CPU offload, peaking at 114.72 GiB vs. LongStraw's 62.94 GiB at 512K, −45.1%-45.1\%), whereas LongStraw completes the 1M update at 98.95 GiB in 18 minutes. Compact storage (for capture) and blockwise reverse replay (for backward) are thus both essential at 1M.
  2. Head-to-head with Prefix Grouper and OOMB (Section 5.2, Table 20): On Qwen2.5-7B-Instruct-1M (one H200), Prefix Grouper runs out of memory at 1M. OOMB's exact (dense) mode (O(P2)O(P^2) chunk backward ×G\times G) is 7.9×7.9\times slower than LongStraw at 128K and had not finished a single 1M update after 28.7 hours; its Sparse64 mode finishes only by approximating attention over 64 pages per chunk. LongStraw is the only system that completes an exact-attention 1M update on one H200 (5,255 s, 69.25 GiB). Moreover, through SPkS_P^k, it spans standard GQA (Qwen2.5-7B-1M), hybrid Gated DeltaNet recurrent/conv + attention (Qwen3.6-27B), and MLA/DSA sparse attention + routed MoE (GLM-5.2 on 32 H200s at 2M).
  3. Why a short tail suffices (91-run sweep at 128K; Section 5.1, Tables 21--22): Across 13 paired seeds and seven scopes b∈{0,1,4,8,32,128,full}b \in \{0, 1, 4, 8, 32, 128, \text{full}\}, b=0b=0 gives 0.00 EM gain in every seed (confirming Proposition 1), whereas b≥8b \ge 8 recovers nearly all of the full-prompt gain (+1.44 at b=8b=8 and +1.12 at b=32,128b=32, 128 vs. +1.52 for full prompt) while saving 14.53 GiB and 36% of GPU time.
  4. Head-to-head with full-prompt training on 1,000-token legal summaries (Multi-LexSum; Section 5.4, Table 25): On 171 civil-rights cases at 64K--128K tokens (near the memory limit of native full-prompt LoRA-RL with CPU offload), Tail32 achieves a higher ROUGE-L gain than native full-prompt training in all three seeds (+0.40 vs. +0.19 mean gain) and a higher final score (25.04) than both full-prompt training (24.87) and 32K retrieval (24.63).
  5. Natural million-token chat histories (LongMemEval_M, Table 24) and multi-seed 2M/4M scaling (Table 23): On LongMemEval_M, each of the 96 held-out questions has its own independent 500-session history (1.01--1.15M tokens). Across three seeds, held-out EM rises from 22.92% to 28.47% (+5.56) and F1 from 32.28% to 40.68% (+8.40, 95% CI [3.48, 13.90]), improving in every seed. Across three seeds each at 2M and 4M tokens (512-token outputs), F1 improves in all six runs (4M mean: +4.51 EM / +3.74 F1).
  6. Multi-step numerical stability (Sections 5.2 and 5.5, Tables 26--27): Over three successive optimizer steps from matched states, the AdamW update relative L2L_2 difference drops from 0.183→0.083→0.0610.183 \rightarrow 0.083 \rightarrow 0.061 (128K) and 0.251→0.152→0.0680.251 \rightarrow 0.152 \rightarrow 0.068 (512K) as second-moment estimates damp step-1 near-zero sign flips; recomputing the GLM-5.2 output-head check in FP64 lowers the group-12 discrepancy from 3.72×10−53.72\times 10^{-5} to 9.49×10−89.49\times 10^{-8} across all 32 ranks.

Full data tables and point-by-point responses are in our replies to each reviewer below. We warmly welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.