- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces R-NOTES, a long-context framework built entirely on sliding-window attention (SWA). The model's self-generated notes are retained within the attention window as dynamic carriers of information, enabling a pure SWA model to process inputs of unbounded length. Trained jointly via SFT and outcome-based GRPO, R-NOTES matches or surpasses far larger full-attention baselines on benchmarks up to 512K tokens, while maintaining a constant KV cache and linearly growing inference compute.
Critical strengths
- R-NOTES dispenses with hybrid attention architectures and external vector databases, recasting long-context memory management as standard autoregressive language generation. It is both elegant and instructive.
- The KV cache is bounded, eliminating OOM risk. Inference FLOPs scale linearly with input length, making R-NOTES attractive for resource-constrained or unbounded streaming deployment.
- Despite fine-tuning only on HotpotQA with sequences up to 32K, it remains robust on needle-in-a-haystack tests at 512K outperforming QwenLong-L1-32B trained with long-context RL.
Critical weaknesses
- Full attention requires a single parallelizable prefill. R-NOTES must generate notes token by token after every chunk (if needed). Latency and wall-clock comparisons against baselines at varying lengths are warranted.
- Since the core idea is chunk-wise compression via self-generated records, the absence of a comparison LLM×MapReduce (also divide-and-conquer and training-free) leaves the efficiency–performance trade-off insufficiently substantiated. [ref] LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Models
- It has no mechanism for look-back. If it deems a detail irrelevant and omits it from the notes, that detail is physically annihilated once it exits the 12K window. No tolerance, no recourse.
- Error accumulation and spurious RL credit. Outcome-only RL applies no fine-grained penalty to hallucinated intermediate notes: fabricated facts that nonetheless yield the correct answer reinforce a spurious correlation. Errors also propagate, once an early note is wrong, subsequent notes inherit and compound it. A case study is needed.
Questions for authors
None
3 comments
Response to "no mechanism for look-back"
We agree that R-Notes reads the input in a single forward pass, so a detail that is not written into a note before it leaves the window cannot be recovered. However, two observations show that this has limited impact in practice. We have added a short discussion of this limitation in Appendix D.3 (highlighted in blue).
- Omission does not compound with length. Every note restates the question, so each keep-or-drop decision is made with the query in view. On RULER-HQA, the gold answer is written into at least one note in 80.5%, 81.3%, 80.5% and 79.7% of samples at 14K, 28K, 56K and 112K (Appendix Table 11). The capture rate stays essentially flat across an 8× increase in input length, so the rate of irrecoverable omission does not grow as more content leaves the window.
- Once captured, information is not lost to eviction. Given capture, the gold answer is still present in the last note in 96–98% of samples at every length (Appendix Table 11), and in more than 92% of samples at every note load in our pooled RULER-HQA and LongBench analysis (Appendix Table 12).
Moreover, other bounded-memory methods share this property. MemAgent overwrites a fixed-size memory at every chunk, and LLM×MapReduce discards chunks whose map output is "[NO INFORMATION]" before the reduce stage; neither can revisit evicted text. Only full attention retains every token, at the cost of a KV cache that grows linearly with input length.
Finally, look-back can be layered on top of R-Notes when the source remains accessible. In the BrowseComp-Plus agent setting (Appendix D.6), the model can re-issue retrieval calls and re-read documents, and R-Notes outperforms Qwen3-32B and Search-R1-32B. We will discuss explicit replay of evicted chunks as a future extension.
Response to "error accumulation and spurious RL credit"
We agree that outcome-only RL gives no direct penalty to unsupported note content. However, our analysis shows that RL does not reinforce unfaithful notes in practice. We have added this analysis and the requested case study in Appendix D.4 (highlighted in blue).
- RL makes notes more faithful, not less. If RL rewarded fabricated notes that happen to yield correct answers, accuracy would rise without the notes carrying the gold evidence more reliably. We observe the opposite. From the SFT checkpoint to the RL checkpoint, capture, conditional retention and accuracy all increase at every length. At 112K, capture rises from 70.3% to 79.7%, conditional retention from 86.7% to 97.1%, and accuracy from 76.6% to 87.5% (Appendix Table 11). The RL gain therefore comes from notes that contain and preserve the gold evidence more often.
- Distortion exists but is bounded. On NarrativeQA, Qasper and MuSiQue, the three LongBench tasks where R-Notes scores lowest, 23% of ground-truth facts appear in distorted form, mostly as over-enumerated methods on Qasper (Figure 9).
Case study of error propagation (Appendix D.4). On NarrativeQA, for the question "What does Louis tell Jacob that hell burns away?" (gold: "your memories and attachments"), the 6th and 7th notes record the correct quote. A later chunk contains a different line by the same character ("If you're frightened of dying…"), and the 8th note overwrites the correct entry with it. Notes 9–12 inherit the wrong entry, and the final answer is wrong. The error comes from a later passage about the same character that the model treats as new evidence, rather than from a fact invented from nothing. Such losses after capture are rare: given capture, the gold answer remains in the final note in 96–98% of RULER-HQA samples (Appendix Table 11).
A process-level reward that verifies note content against the source chunk could penalize such updates directly. We will discuss this as future work in the revision.
Dear Reviewer,
Thank you for your review and helpful comments. We have revised the manuscript to address the main concerns. Specifically, we added end-to-end latency and wall-clock comparisons in Appendix D.8 (Table 19); a same-backbone comparison with LLM×MapReduce in Appendix D.7 (Tables 17–18); a discussion of the look-back limitation in Appendix D.3; and an analysis of error accumulation and spurious RL credit, including the requested case study, in Appendix D.4. All changes are highlighted in blue in the revised appendix.
As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.
Best regards,
The authors
Response to "latency and wall-clock comparisons"
We add end-to-end latency measurements comparing R-Notes with the Qwen3-8B full-attention baseline across five input lengths on the same hardware. We have added these results in Appendix D.8 (Table 19, highlighted in blue).
Table 1. End-to-end latency (seconds) across input lengths.
R-Notes is faster than the Qwen3-8B full-attention baseline at every input length, and the speedup grows with length, from 1.92× at 14K to 3.13× at 224K. The additional decode load from generating notes does not offset the compute savings of pure SWA. This confirms that the FLOPs advantage of R-Notes translates into a real end-to-end latency advantage.
Response to "comparison with LLM×MapReduce"
Thank you for the suggestion. We have added a comparison with LLM×MapReduce in Appendix D.7 (Tables 17 and 18, highlighted in blue). We use its official implementation with unmodified prompts and the default chunk size of 4K, with the same Qwen3-8B backbone as R-Notes and the same RULER-HQA set.
Table A. Accuracy (%) on RULER-HQA.
Table B. Generated tokens per sample on RULER-HQA.
Performance. R-Notes outperforms LLM×MapReduce by 28.5 points on average (Table A). Most LLM×MapReduce failures occur when the evidence for a multi-hop question is spread across different chunks. Each map call sees only one chunk, and chunks whose map output is "[NO INFORMATION]" are discarded before the reduce stage. The reduce stage therefore often cannot connect the evidence and answers that the question cannot be determined. R-Notes instead carries partial evidence forward in its notes, so later chunks can complete the chain.
Efficiency.