acceptodds
Under review as a conference paper at ICLR 2027

R-Notes: Sliding-Window LLMs Can Learn to Remember Beyond Their Attention Window

Abstract

Real-world LLM deployments increasingly require processing inputs of effectively unbounded length, calling for models that can handle arbitrarily long inputs with bounded memory. However, existing constant-memory architectures, such as linear attention and pure sliding window attention (SWA), lose information beyond their fixed state or window. To address this, we observe that an SWA model can preserve information much like a reader taking notes: brief self-written notes stay in the recent window and carry key content forward, even as the original tokens scroll out. Trained to do this consistently, pure SWA models already suffice for lengthy inputs. Building on this observation, we propose R-Notes, which feeds fixed-size chunks of the input as a multi-turn conversation under a single SWA session and trains the model to emit brief notes that propagate key information across chunks within the window. Built on Qwen3-8B, R-Notes matches or exceeds the 1.75x larger Qwen3-14B full-attention baseline on LongBench cross-task QA (73.4% vs 71.3%) and on the longest settings of RULER-HQA at 112K and RULER-SQuAD at 128K (87.50% vs 78.12% on both), while keeping the KV cache bounded by the window size and total compute linear in the number of processed chunks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Long context, Sliding window attention, Memory management

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jqqcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces R-NOTES, a long-context framework built entirely on sliding-window attention (SWA). The model's self-generated notes are retained within the attention window as dynamic carriers of information, enabling a pure SWA model to process inputs of unbounded length. Trained jointly via SFT and outcome-based GRPO, R-NOTES matches or surpasses far larger full-attention baselines on benchmarks up to 512K tokens, while maintaining a constant KV cache and linearly growing inference compute.

Critical strengths
  1. R-NOTES dispenses with hybrid attention architectures and external vector databases, recasting long-context memory management as standard autoregressive language generation. It is both elegant and instructive.
  2. The KV cache is bounded, eliminating OOM risk. Inference FLOPs scale linearly with input length, making R-NOTES attractive for resource-constrained or unbounded streaming deployment.
  3. Despite fine-tuning only on HotpotQA with sequences up to 32K, it remains robust on needle-in-a-haystack tests at 512K outperforming QwenLong-L1-32B trained with long-context RL.
Critical weaknesses
  1. Full attention requires a single parallelizable prefill. R-NOTES must generate notes token by token after every chunk (if needed). Latency and wall-clock comparisons against baselines at varying lengths are warranted.
  2. Since the core idea is chunk-wise compression via self-generated records, the absence of a comparison LLM×MapReduce (also divide-and-conquer and training-free) leaves the efficiency–performance trade-off insufficiently substantiated. [ref] LLM×MapReduce: Simplified Long-Sequence Processing using Large Language Models
  3. It has no mechanism for look-back. If it deems a detail irrelevant and omits it from the notes, that detail is physically annihilated once it exits the 12K window. No tolerance, no recourse.
  4. Error accumulation and spurious RL credit. Outcome-only RL applies no fine-grained penalty to hallucinated intermediate notes: fabricated facts that nonetheless yield the correct answer reinforce a spurious correlation. Errors also propagate, once an early note is wrong, subsequent notes inherit and compound it. A case study is needed.
Questions for authors

None

3 comments

Response to "latency and wall-clock comparisons"

We add end-to-end latency measurements comparing R-Notes with the Qwen3-8B full-attention baseline across five input lengths on the same hardware. We have added these results in Appendix D.8 (Table 19, highlighted in blue).

Table 1. End-to-end latency (seconds) across input lengths.

Method14K28K56K112K224K
R-Notes6.46.910.922.641.0
Qwen3-8B12.310.714.649.0128.3
Speedup1.92×1.55×1.34×2.17×3.13×

R-Notes is faster than the Qwen3-8B full-attention baseline at every input length, and the speedup grows with length, from 1.92× at 14K to 3.13× at 224K. The additional decode load from generating notes does not offset the compute savings of pure SWA. This confirms that the FLOPs advantage of R-Notes translates into a real end-to-end latency advantage.

Response to "comparison with LLM×MapReduce"

Thank you for the suggestion. We have added a comparison with LLM×MapReduce in Appendix D.7 (Tables 17 and 18, highlighted in blue). We use its official implementation with unmodified prompts and the default chunk size of 4K, with the same Qwen3-8B backbone as R-Notes and the same RULER-HQA set.

Table A. Accuracy (%) on RULER-HQA.

Method14K28K56K112KAvg.
LLM×MapReduce (Qwen3-8B)64.8458.5960.1661.7261.33
R-Notes (Qwen3-8B)92.9789.8489.0687.5089.84

Table B. Generated tokens per sample on RULER-HQA.

Method14K28K56K112K
LLM×MapReduce (Qwen3-8B)1.4K2.3K4.1K7.6K
R-Notes (Qwen3-8B)0.3K0.5K0.8K1.4K

Performance. R-Notes outperforms LLM×MapReduce by 28.5 points on average (Table A). Most LLM×MapReduce failures occur when the evidence for a multi-hop question is spread across different chunks. Each map call sees only one chunk, and chunks whose map output is "[NO INFORMATION]" are discarded before the reduce stage. The reduce stage therefore often cannot connect the evidence and answers that the question cannot be determined. R-Notes instead carries partial evidence forward in its notes, so later chunks can complete the chain.

Efficiency.

  1. Sparse note emission. LLM×MapReduce generates extracted information, a rationale, an answer and a confidence score for every chunk. R-Notes writes a note only when the chunk contains relevant information. On RULER-HQA, it stays silent in about two-thirds of chunk turns: it emits 17,546 notes across 1,536 trajectories, which contain roughly 50K chunk turns (Appendix D.2). Its notes average 97 tokens (Appendix Table 8), and the total note stream amounts to only 1.2–2.1% of the input (Appendix Table 9). As a result, R-Notes generates about 5× fewer tokens at every length (Table B).
  2. No re-prefill. LLM×MapReduce issues an independent call for every chunk (45 calls per sample at 112K), and its collapse and reduce stages re-encode the map outputs as new prompts. R-Notes processes the input in a single continuous SWA session, where earlier notes are already in the KV cache and are never re-encoded.

Response to "no mechanism for look-back"

We agree that R-Notes reads the input in a single forward pass, so a detail that is not written into a note before it leaves the window cannot be recovered. However, two observations show that this has limited impact in practice. We have added a short discussion of this limitation in Appendix D.3 (highlighted in blue).

  1. Omission does not compound with length. Every note restates the question, so each keep-or-drop decision is made with the query in view. On RULER-HQA, the gold answer is written into at least one note in 80.5%, 81.3%, 80.5% and 79.7% of samples at 14K, 28K, 56K and 112K (Appendix Table 11). The capture rate stays essentially flat across an 8× increase in input length, so the rate of irrecoverable omission does not grow as more content leaves the window.
  2. Once captured, information is not lost to eviction. Given capture, the gold answer is still present in the last note in 96–98% of samples at every length (Appendix Table 11), and in more than 92% of samples at every note load in our pooled RULER-HQA and LongBench analysis (Appendix Table 12).

Moreover, other bounded-memory methods share this property. MemAgent overwrites a fixed-size memory at every chunk, and LLM×MapReduce discards chunks whose map output is "[NO INFORMATION]" before the reduce stage; neither can revisit evicted text. Only full attention retains every token, at the cost of a KV cache that grows linearly with input length.

Finally, look-back can be layered on top of R-Notes when the source remains accessible. In the BrowseComp-Plus agent setting (Appendix D.6), the model can re-issue retrieval calls and re-read documents, and R-Notes outperforms Qwen3-32B and Search-R1-32B. We will discuss explicit replay of evicted chunks as a future extension.

Response to "error accumulation and spurious RL credit"

We agree that outcome-only RL gives no direct penalty to unsupported note content. However, our analysis shows that RL does not reinforce unfaithful notes in practice. We have added this analysis and the requested case study in Appendix D.4 (highlighted in blue).

  1. RL makes notes more faithful, not less. If RL rewarded fabricated notes that happen to yield correct answers, accuracy would rise without the notes carrying the gold evidence more reliably. We observe the opposite. From the SFT checkpoint to the RL checkpoint, capture, conditional retention and accuracy all increase at every length. At 112K, capture rises from 70.3% to 79.7%, conditional retention from 86.7% to 97.1%, and accuracy from 76.6% to 87.5% (Appendix Table 11). The RL gain therefore comes from notes that contain and preserve the gold evidence more often.
  2. Distortion exists but is bounded. On NarrativeQA, Qasper and MuSiQue, the three LongBench tasks where R-Notes scores lowest, 23% of ground-truth facts appear in distorted form, mostly as over-enumerated methods on Qasper (Figure 9).

Case study of error propagation (Appendix D.4). On NarrativeQA, for the question "What does Louis tell Jacob that hell burns away?" (gold: "your memories and attachments"), the 6th and 7th notes record the correct quote. A later chunk contains a different line by the same character ("If you're frightened of dying…"), and the 8th note overwrites the correct entry with it. Notes 9–12 inherit the wrong entry, and the final answer is wrong. The error comes from a later passage about the same character that the model treats as new evidence, rather than from a fact invented from nothing. Such losses after capture are rare: given capture, the gold answer remains in the final note in 96–98% of RULER-HQA samples (Appendix Table 11).

A process-level reward that verifies note content against the source chunk could penalize such updates directly. We will discuss this as future work in the revision.

Dear Reviewer,

Thank you for your review and helpful comments. We have revised the manuscript to address the main concerns. Specifically, we added end-to-end latency and wall-clock comparisons in Appendix D.8 (Table 19); a same-backbone comparison with LLM×MapReduce in Appendix D.7 (Tables 17–18); a discussion of the look-back limitation in Appendix D.3; and an analysis of error accumulation and spurious RL credit, including the requested case study, in Appendix D.4. All changes are highlighted in blue in the revised appendix.

As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards,

The authors

Reviewer_qyv4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

R-NOTES trains a Qwen3-8B model to process long inputs as chunks under pure sliding-window attention (SWA). After each chunk, the model can either emit a short note that carries task-relevant information forward or remain silent. The method uses supervised fine-tuning on teacher-generated note-taking trajectories, followed by GRPO with final-answer correctness as the reward. The authors evaluate R-NOTES on LongBench, RULER, and NIAH, reporting competitive long-context QA performance, including results at up to 128K tokens on RULER and 512K on NIAH. They also analyze accuracy, note propagation, and inference efficiency, arguing that the approach maintains a bounded KV cache without an external memory module or an expanded attention window.

Critical strengths

R-NOTES provides a simple way to carry task-relevant information across chunks while retaining a fixed sliding-attention window. The method does not require a separate memory module or a larger context window.

Critical weaknesses

The experiments do not convincingly isolate R-NOTES’ advantage over existing external-memory or agent-based long-context methods. The comparison in this category is primarily with MemAgent, and the paper does not provide a same-backbone controlled comparison or head-to-head latency and other end-to-end cost measurements. Consequently, the results do not establish whether the proposed mechanism is more effective or efficient than this broader class of methods.

Questions for authors

Across different sliding-window sizes, what proportion of evaluation trajectories satisfy the refresh condition and preserve the answer-relevant information required by the coverage condition? How does performance differ between trajectories that satisfy these conditions and those that violate them?

3 comments

Response to "isolating the advantage over external-memory and agent-based methods"

Thank you for this comment. To isolate the advantage of R-Notes on the same backbone, we have added a comparison with LLM×MapReduce, a training-free divide-and-conquer long-context method, in Appendix D.7 (Tables 17 and 18, highlighted in blue). We use its official implementation with unmodified prompts and the default chunk size of 4K, with the same Qwen3-8B backbone as R-Notes. Like MemAgent, it processes the input through many independent model calls and keeps intermediate state outside the model (the per-chunk map outputs), placing it in the class of external-state methods the reviewer refers to.

Table A. Accuracy (%) on RULER-HQA with the Qwen3-8B backbone (Appendix Table 17).

Method14K28K56K112KAvg.
LLM×MapReduce (Qwen3-8B)64.8458.5960.1661.7261.33
R-Notes (Qwen3-8B)92.9789.8489.0687.5089.84

Table B. Generated tokens per sample on RULER-HQA (Appendix Table 18).

Method14K28K56K112K
LLM×MapReduce (Qwen3-8B)1.4K2.3K4.1K7.6K
R-Notes (Qwen3-8B)0.3K0.5K0.8K1.4K

R-Notes is more effective. With the same backbone, R-Notes outperforms LLM×MapReduce by 28.5 points on average and at every length (Table A). Most LLM×MapReduce failures occur when the evidence for a multi-hop question is spread across different chunks. Each map call sees only one chunk, and chunks whose map output is "[NO INFORMATION]" are discarded before the reduce stage, so the reduce stage often cannot connect the evidence. R-Notes instead carries partial evidence forward in its notes, so later chunks can complete the reasoning chain.

R-Notes is more efficient.

  1. Fewer generated tokens. LLM×MapReduce generates extracted information, a rationale, an answer and a confidence score for every chunk. R-Notes writes a note only when a chunk contains relevant information and stays silent in about two-thirds of chunk turns (Appendix D.2). Its notes average 97 tokens (Appendix Table 8). As a result, R-Notes generates about 5× fewer tokens at every length (Table B).
  2. No re-encoding. LLM×MapReduce issues an independent model call for every chunk (45 calls per sample at 112K), and its collapse and reduce stages re-encode the map outputs as new prompts. R-Notes processes the whole input in a single continuous SWA session: earlier notes are already in the KV cache and are never re-encoded, and the KV cache stays bounded by the 12K window regardless of input length.

Response to "refresh and coverage across window sizes"

Thank you for this question. We answer it directly from the saved evaluation trajectories of the window-size ablation (Table 4). For each window, we analyze 512 RULER-HQA trajectories (128 at each of 14K, 28K, 56K and 112K) and measure the two conditions as follows. We have added the analysis as Appendix D.9 (Table 20, highlighted in blue).

  • Refresh. We check every hop of the note chain: from the query to the first note, from each note to the next, and from the last note to the final answer. For each hop, we count the actual tokens in the dialogue between the first token of the earlier item and the last token of the later one, and the hop satisfies refresh if this span fits within the window WW. We report the share of hops that satisfy refresh.
  • Coverage. Whether the gold answer appears in the last non-empty note before the final answer, using the same normalization as our capture/retain analysis (Appendix D.3). This is a conservative proxy: it misses paraphrases of the gold answer and does not credit bridge entities in multi-hop questions.

Table C. Share of note-chain hops satisfying refresh (%), share of trajectories satisfying coverage (%), and overall accuracy (%).

WindowRefresh satisfied (hops)Coverage satisfied (trajectories)Overall acc.
2K (CC=256)904252.3
8K99.77486.7
12K (default)10077.989.8
16K10077.989.7

We draw three conclusions.

  1. For W≥8W \ge 8K, refresh holds. Because the chunk size is chosen together with the window so that Eq. 1 holds, the learned schedule keeps the chain refreshed on essentially every hop. At the default setting, a hop spanning the suggested interval of four chunks takes about 7.3K tokens, well within the 12K window.
  2. Accuracy follows coverage. At every window, trajectories whose last note carries the gold answer reach 96–97.5% accuracy, whereas the remaining trajectories reach 21–64%. Coverage is therefore the condition that determines success in practice. At W≥8W \ge 8K, the coverage-violated subset still reaches 57–64% accuracy, which largely reflects the limits of the substring proxy, such as yes/no comparison questions whose answer is never written verbatim, or aliases of the gold entity. The 3.1-point accuracy drop from 12K to 8K is accompanied by a 3.9-point drop in coverage (77.9% to 74%).
  3. At WW = 2K, a few hops overshoot slightly, but the collapse is caused by coverage.
    • About 10% of hops exceed the 2K window, by fewer than 50 tokens on average. Such small overshoots have little effect. Notes are copied sequentially: the tokens pushed out of the window are the first few tokens of the earlier note, typically the restated question, which the new note has already reproduced by then. More precisely, the kk-th token of Ni\mathcal{N}_i only needs to remain visible while the kk-th token of Ni+1\mathcal{N}_{i+1} is written, which requires roughly nC+Lnotes≤WnC + L_{\text{notes}} \le W; this holds on more than 99% of hops.

    • Coverage is what collapses. A 112K input requires about 6× more note-to-note hops at 2K than at 12K, and small per-hop losses compound, so coverage falls to 42%. Trajectories that do retain the answer still reach 96% accuracy, the same level as at larger windows. This confirms the explanation in §4.3: refresh keeps the chain alive, while a too-small window overloads coverage.

Overall, the learned note schedule satisfies refresh across window sizes, and the remaining errors come mainly from coverage, which is consistent with the capture/retain analysis in Appendix D.3.

Dear Reviewer,

Thank you for your review and helpful questions. We have revised the manuscript to address the main concerns. First, we added a same-backbone comparison with LLM×MapReduce in Appendix D.7 (Tables 17–18). Second, we added an analysis of 512 RULER-HQA trajectories across sliding-window sizes in Appendix D.9 (Table 20).

As the discussion period nears its end, we would like to follow up on our author response. We would appreciate any further feedback and would be happy to clarify any remaining concerns.

Best regards,

The authors

Reviewer_KSqSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes R-NOTES, a method that lets a sliding-window attention (SWA) model handle long-context question answering through chunk-wise processing and interleaved notes. It trains the model with SFT and RL. Extensive experiments show that SWA can achieve performance comparable to self-attention models.

Critical strengths
  1. The problem is important and meaningful.
  2. The method is clearly presented. The paper explains chunk-wise processing and the interleaved note mechanism well.
  3. The experiments on RULER and LongBench are extensive and show the effectiveness of the method.
Critical weaknesses
  1. The paper lacks a detailed comparison with related work. Smooth Reading [1] uses a similar approach for long-context understanding with recurrent LLMs, including sliding-window attention. The differences are mainly in the note mechanism and training strategy (SFT and RL). A more thorough comparison would better highlight the novelty and advantages of R-NOTES.

[1] Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Understanding (ICLR 2026)

Questions for authors

see weakness

3 comments

Thank you for pointing us to Smooth Reading. We now cite and discuss it in the Context management paragraph of Section 2 (Related Work), with the newly added discussion highlighted in blue. As its trained checkpoint is not publicly available, we cannot directly compare against the authors' trained model. Nevertheless, the two approaches differ in their memory-update mechanisms and training procedures.

Sparse, adaptive note emission. Smooth Reading generates a structured summary after each chunk, updating accumulated clues and documenting the rationale for the updates. In contrast, R-NOTES learns whether and when to emit a note, rather than generating a memory update for every chunk. We have evaluated the benefit of adaptive emission in a controlled ablation (Figure 8). Under the same experimental conditions, adaptive emission outperforms forced emission on all three evaluated benchmarks. These results support learning when to write a note rather than requiring an update after every chunk.

Compression and token efficiency. In our experiments, R-NOTES emits no note in approximately two-thirds of chunk-processing rounds. By avoiding note generation when a memory update is unnecessary, R-NOTES can achieve stronger compression and better token efficiency.

Training and generalization. Smooth Reading is trained with supervised fine-tuning (SFT) on 48,856 examples. In contrast, R-NOTES combines SFT on approximately 3,000 examples with reinforcement learning (RL) using approximately 1,000 examples. Despite using substantially fewer training examples, R-NOTES generalizes to diverse English and Chinese QA datasets. Its transfer also extends beyond QA to summarization, few-shot in-context learning, synthetic retrieval, and the BrowseComp-Plus deep-research agent task. These results suggest that R-NOTES achieves higher training sample efficiency while learning a note-taking policy that transfers across tasks, languages, and application settings.

Thanks for your response.

I acknowledge the contribution of the paper. Due to sliding window LLM with context mamnagement is a promising direction and meaningful to the community, I'm willing to raise my score.

Thank you very much for your positive follow-up and for raising your score. We greatly appreciate your recognition of the paper’s contribution and of this promising direction for the community.

Comments on the paper

1 comment

Dear Reviewers,

We would like to thank all reviewers for their constructive feedback. We are encouraged that the reviewers recognize our paper:

  • has proposed "an important and meaningful" problem (Reviewer KSqS), with a method that is "clearly presented" and whose experiments are "extensive and show the effectiveness of the method" (Reviewer KSqS).
  • provides "a simple way to carry task-relevant information across chunks while retaining a fixed sliding-attention window" that "does not require a separate memory module or a larger context window" (Reviewer qyv4).
  • "dispenses with hybrid attention architectures and external vector databases, recasting long-context memory management as standard autoregressive language generation" in a way that is "both elegant and instructive" (Reviewer jqqc), with a bounded KV cache and linear inference cost.

We have been working diligently on improving the paper in several aspects, addressing your concerns and questions. Below, we summarise the main improvements, all of which have been included in the revised version (highlighted in blue).

1. Comparison with Smooth Reading (Section 2, Figure 8)

We now cite and discuss Smooth Reading in Section 2 (Related Work), highlighting the key differences: R-Notes learns adaptive, sparse note emission (silent in ~2/3 of chunk turns) that outperforms forced emission across all three evaluated benchmarks (Figure 8), and generalizes across English/Chinese QA, summarization, few-shot ICL, synthetic retrieval, and BrowseComp-Plus using ~3,000 SFT + ~1,000 RL examples (vs. 48,856 SFT examples).

2. Same-backbone comparison with LLM×MapReduce (Appendix D.7, Tables 17–18)

Using the same Qwen3-8B backbone on RULER-HQA (14K–112K, default 4K chunks), R-Notes outperforms LLM×MapReduce by 28.5 points on average (89.84% vs. 61.33%) because it carries partial multi-hop evidence forward instead of discarding [NO INFORMATION] chunks. It also generates ~5× fewer tokens (0.3K–1.4K vs. 1.4K–7.6K per sample) in a single continuous SWA session, avoiding 45 independent calls per sample at 112K and prompt re-encoding.

3. End-to-end latency measurements (Appendix D.8, Table 19)

Measured on the same hardware across 14K–224K tokens, R-Notes is faster than the Qwen3-8B full-attention baseline at every length, with speedup growing from 1.92× at 14K (6.4s vs. 12.3s) to 3.13× at 224K (41.0s vs. 128.3s). Note decoding does not offset the compute savings of pure SWA.

4. Refresh and coverage analysis across window sizes (Appendix D.9, Table 20)

Analyzing 512 saved RULER-HQA trajectories from Table 4, refresh holds on 99.7–100% of hops at ≥8K windows, and accuracy is governed by coverage (96–97.5% when the final note retains the gold answer vs. 21–64% otherwise). At 2K, ~6× more hops compound per-hop losses and drop coverage to 42% (52.3% overall accuracy), though trajectories that retain the answer still reach 96% accuracy.

5. Discussion of the look-back limitation (Appendix D.3, Tables 11–12)

We acknowledge that single-pass reading cannot recover details omitted before leaving the window, and show its limited practical impact: RULER-HQA capture rate stays flat across 14K–112K (79.7–81.3%), and conditional retention in the final note reaches 96–98% across lengths (>92% across note loads).

6. Error accumulation and spurious RL credit analysis (Appendix D.4, Table 11, Figure 9)

From SFT to RL at 112K, capture (70.3% → 79.7%), conditional retention (86.7% → 97.1%), and accuracy (76.6% → 87.5%) all rise together, refuting spurious RL credit. Distortion on the three lowest-scoring LongBench tasks is bounded (23%, mainly over-enumerated methods on Qasper), and our NarrativeQA case study shows rare post-capture overwrites from later passages about the same character.

We thank the reviewers again for their insightful feedback, which has substantially improved the rigor, completeness, and clarity of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.