- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes OSET (Online Sublinear Evidence Tree), which addresses a problem largely overlooked by existing long-context methods in online settings: constraining the amount of full-text state retained by the system to grow sublinearly with the input length while minimizing the loss of critical evidence needed for future queries.
Critical strengths
-
The paper defines a clear and practically relevant research problem. The proposed setting lies between fixed-capacity online memory and linearly growing external memory. It imposes stricter constraints than approaches that focus solely on KV-cache compression or retrieval, making it more representative of real-world streaming scenarios.
-
The separation of semantic access from source evidence is a reasonable design choice. Summarization can lose precise information, such as entities and numerical values, while retaining only the original text leads to linear memory growth. OSET addresses this trade-off by storing semantic cues and supporting evidence (e.g., entities and values) separately, while accounting for both within the memory budget. This is a pragmatic approach to balancing retrieval efficiency and evidence preservation.
-
The evaluation goes beyond end-task performance. The authors examine whether evidence is preserved, successfully retrieved, and effectively used by the reader, allowing them to distinguish memory failures from reasoning failures. This provides a more diagnostic analysis than relying solely on end-task metrics such as F1.
Critical weaknesses
-
The benefits of the sublinear constraint are not robust. In their paper, the budget is still considerable at the 128K level. It proves that "it can work under this budget," but it doesn't fully demonstrate "how much sublinearity saves compared to linear growth or fixed capacity, and where the savings lie." The reader context has a maximum of 8 independent nodes, but the actual token occupancy curve of the persistent state is not clearly shown.
-
The results show obvious problems. RULER's (FWE) dropped from 97.56 to 30.07, and the QA task also saw a significant decline. This means that OSET's applicability is narrower than the title suggests. It is more like a "sparse evidence retrieval-type long context" method than a general long context memory.
-
Again, the results for multi-hop are not robust. On HotpotQA, OSET scores of 61.0 are higher than those of full context scores of 58.0, which is somewhat counterintuitive, possibly because the search context reduces interference; however, scores for MuSiQue (31.0 vs 36.0) and 2Wiki (43.0 vs 45.0) both decline. Overall, robustness in multi-hop scenarios has not yet been fully established.
Questions for authors
NA
2 comments
Thanks for your response; I will maintain my initial rating.
R3.1 — Practical storage savings from sublinear retention
The actual RULER occupancy trajectory (Table R3) shows source length increasing 16.9× between 8K and 128K, while retained active content increases only 3.7×. Correspondingly, active-content occupancy falls from 38.93% to 8.60%; its descriptive finite-range log–log slope is 0.468 (Table S3). At 2M input tokens, the additional streaming experiment retains 2.41% of source length in active textual memory (Table S7). These measurements demonstrate progressively selective active evidence retention; complete serialized-state accounting, including the metadata-reclamation issue, is reported separately in R2.Q2.
R3.2 — Global aggregation and multiple-evidence tasks
OSET targets sparse, verifiable evidence and inter-evidence dependencies, while frequent-word extraction (FWE) requires global aggregation over many source locations and has a different information requirement. We also clarify a numerical mix-up in the review: submitted Table 3 reports FWE 83.60 → 30.07, whereas 97.56 is OSET's variable-tracking (VT) result, not the full-context FWE score.
Multi-evidence access remains meaningful under constrained memory: at 128K, OSET obtains 78.45 on multi-value retrieval, 76.20 on multi-query retrieval, and 42.40 on HotpotQA-based multi-hop QA2 (submitted Table 3). Together with the p(n)k illustration in R1.2, these observations support targeted evidence retention as a valuable alternative to indiscriminate selection, while identifying corpus-wide counting as outside the method's principal design target.
R3.3 — Multi-hop OpenQA and learned retrieval
OpenQA tests access to complementary evidence beyond direct lexical matches. In the submitted results, OSET improves HotpotQA over full context (61.0 vs 58.0 F1), while MuSiQue (31.0 vs 36.0) and 2Wiki (43.0 vs 45.0) are more challenging. The additional memory-system comparisons show OSET's highest macro F1 among the tested adapted baselines (45.00 vs 44.83, 41.36, and 38.18; Table R1). Under the separate main-score retriever comparison, learned retrieval improves F1 over frozen cosine by 3.05, 3.60, and 0.83 points, respectively (+2.49 macro F1; Table R4). These results support OSET's evidence-oriented organization and the value of learned semantic access for multi-hop reasoning under that setting.