- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Periscope, a training-free method for long-context question answering and retrieval with a frozen LLM. The chunks of a document are arranged on a grid (), and the model is asked the same question over local probes (consecutive chunks) and strided probes (one chunk from each row). Each answer is scored by the sum of its best local and best strided log-odds, and combining the two probe scores of each chunk gives an evidence map at no extra cost. The map is used either to answer directly or to select the top- chunks for one final read. Experiments use Qwen3.5-4B/27B and Gemma-4-26B-A4B on LongBench v2, InfiniteBench and BRIGHT.
Critical strengths
-
Simple and clearly formulated. The probing scheme, composition rule (Eq. 3) and evidence map (Eq. 4) are easy to follow and require no training.
-
Benefits at extreme lengths. Every probe is about tokens, so peak KV cache stays small: the 27B reads 4.5M-token contexts on one 80GB GPU with 3.1 GB of cache per probe versus 296 GB for a single read (Sec. 4.6). It also helps when the evidence falls outside the window, e.g. +4.8 points over the best window read on InfiniteBench with the 4B (82.1 vs. 77.3, Table 2).
Critical weaknesses
1. Restrictive problem formulation, limited relevance, and weakly justified 2D probing. The paper largely reduces long-context QA to locating a small set of supporting chunks: each answer is scored by its single best local and single best strided span (Eq. 3), on the stated premise that one strong span is enough. This does not capture tasks whose evidence is distributed across the text, and the results reflect it: on LongBench v2 multi-document QA, the selected read trails the window read for both Qwen models, 33.6 vs. 40.0 (4B) and 44.8 vs. 53.6 (27B) (Table 6). It is therefore unclear to what extent the method addresses long-context reasoning rather than a restricted evidence-retrieval problem. The practical relevance is also limited for LLM agents, where a long context is typically kept as an external file or store that the model accesses on demand through tool calls or iterative retrieval (e.g., [1, 2, 3]); such access handles both extreme lengths and distributed evidence without a single bounded read. The paper neither discusses nor compares against this setting: its related work covers parallel and chained chunk reading (Chain of Agents, LongAgent) but not interactive, on-demand access. The 2D decomposition is also largely heuristic: the document has no inherent 2D structure, and a high together with a high does not establish that their intersection, cell , contains the evidence, since the two spans can be supported by different chunks. The paper offers no analysis of this assumption beyond the one-line premise.
2. Unconvincing accuracy–compute tradeoff and incomplete validation of the design. The probes process every token twice, and the compute crossover is about 119k tokens for the 4B and 278k for the 27B (Sec. 4.6). On LongBench v2 the selected read only matches the 4B's best window read (41.9 vs. 41.9) at 4.6 vs. 1.2 PFLOPs, and for the 27B it trails the 262k window read (52.7 vs. 55.9) at 30.2 vs. 10.8 PFLOPs (Table 1). The only retrieval baseline on LongBench v2 is single-shot embedding top-; stronger retrieval-based or adaptive test-time strategies at matched compute (larger read budgets, reranking, iterative retrieval) are missing. The ablations (Table 4: local-only vs. local + strided, Chain of Agents; Table 7: selected-chunk budgets) do not isolate the value of the 2D decomposition: there is no comparison with alternative probe layouts or aggregation rules (e.g., sum or soft aggregation instead of max). The evidence map is shown qualitatively (Fig. 4) and assessed indirectly through downstream accuracy and agreement with the whole read (Table 9), but never against ground-truth evidence locations. Overall, the experimental analysis feels incomplete relative to the paper's claims.
Minor:
- The related work section is long relative to the technical contribution and could be substantially condensed.
- The writing leans on stylized phrasing and metaphors that obscure rather than clarify, e.g., the last sentence of the abstract (a GPU that holds the model, not one that holds the text).
- Fig. 2 is a rudimentary schematic and could convey the probing and map construction more informatively.
References
[1] K.-H. Lee et al. A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts. ICML 2024.
[2] H. Chen et al. Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading. arXiv 2023.
[3] H. Trivedi et al. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. ACL 2023.
Questions for authors
-
(Clarification) How often does the peak of the map coincide with the annotated evidence location? Can you report localization accuracy against ground-truth evidence, and how often the maximizing local and strided spans for the predicted answer are supported by different chunks?
-
(Suggestion) At the same probe budget, how does the local/strided layout compare with alternatives (e.g., random or overlapping partitions, other grid shapes), and how does max-aggregation compare with sum or soft aggregation?
-
(Suggestion) Can you compare against retrieval-based, iterative or agentic baselines matched in PFLOPs, e.g., embedding or BM25 retrieval with reranking or a larger read budget, iterative retrieval, or an agent that reads the document on demand (e.g., ReadAgent or MemWalker)? In what settings would Periscope be preferable to such on-demand access?
-
(Clarification) Is there a natural extension to distributed evidence, such as selecting multiple peaks per answer, and would it close the multi-document QA gap in Table 6?
2 comments
Rebuttal
Compute and when Periscope is preferable (W2, Q3). Periscope targets the Pareto frontier of memory and accuracy at every text length, and on long texts the frontier of memory, compute, speed and accuracy together. It reaches the accuracy of the best window read with 1.6 GB of KV cache per call against 4.3 GB (Table 1), unlocking long contexts for small GPUs. On short texts it pays compute for this, about twice that of a single read, since the probes process every token twice, as the reviewer notes. Beyond the crossover of Sec. 4.6 it needs less compute than a single read of the same text, and it is also faster, as the table shows. It gives the complete time per question of both, by text length, with Qwen3.5-4B on one A100.
| Text length (tokens) | 16k | 32k | 65k | 131k | 262k | 524k | 996k |
|---|---|---|---|---|---|---|---|
| Single read (s) | 0.8 | 1.9 | 4.7 | 13.0 | 40.3 | 138.0 | 458.6 |
| Periscope select, probes and final read (s) | 1.9 | 4.4 | 7.1 | 15.3 | 26.7 | 59.0 | 110.1 |
So Periscope is preferable when you have a long text, a large backbone or a small GPU and want results comparable to a full read. On-demand access is a different step: the agent still has to find which passage to open, with a retriever or with generated notes, one call after another. Periscope can be that retriever: its map ranks the chunks of the text, and reading the chunks it picks is 6 points more accurate than reading those picked by BM25 or embedding retrieval.
Distributed evidence (W1, Q4). A deeper read is the natural extension. It recovers part of the gap but does not close it. The selected read already takes the K highest cells of the map, so several peaks are read, and the table gives the accuracy on the one-fact and two-fact questions of BABILong for reads of K, 2K and 4K chunks from the same map.
| Read | One fact (%) | Two facts (%) | Two facts: both among the chunks read (%) | Tokens in the read |
|---|---|---|---|---|
| Best window read | 44.0 | 32.4 | ||
| Periscope select, K chunks | 66.0 | 22.4 | 16.4 | 9.7k |
| Periscope select, 2K chunks | 65.2 | 29.2 | 30.4 | 19.4k |
| Periscope select, 4K chunks | 56.0 | 26.4 | 48.4 | 38.8k |
A read of 2K chunks holds both facts almost twice as often and gains 6.8 points on two-fact questions, with the one-fact accuracy unchanged. A read of 4K chunks holds both facts more often still and loses 10 points on one-fact questions, because the read loses precision as it grows. The remaining gap points to how recognizable a piece of evidence is from its span, since a probe sees it with its neighbours alone and not with the full text, so a fact that does not decide the answer on its own is harder to recognize as evidence. The multi-document questions of Table 6 have this property by design, since LongBench v2 requires their answers to be drawn from several documents, and there a deeper read does not move the accuracy (33.6% from K/2 to 4K chunks). Multi-document QA is also the only domain of that table where the selected read is behind the window read with both backbones, and the agent the reviewer cites is behind there as well: ReadAgent scores 36.8%, against 40.0% for the window read. We recognize this as a current limitation of Periscope and will add it to the limitations section.
Minor points. We will condense the related work, rewrite the stylized sentences in plain terms, and improve Figure 2 to make it more informative.
Rebuttal
We thank the reviewer for the constructive comments. We answer each point below.
Localization against ground-truth evidence (W1, W2, Q1). We measured it with Qwen3.5-4B on the 750 BABILong questions, which have one, two or three supporting facts at known positions in texts of 32k, 131k and 1M tokens. The last column counts the case the reviewer describes: a peak with no supporting fact whose local span and strided span are each supported by a different chunk.
With one supporting fact the map localizes far above chance, where a random read of K chunks holds the fact in 2% to 14% of the cases, depending on the length. A peak whose two spans are supported by different chunks is rare, 3.6% and 2.4% of the questions with several facts.
Alternative layouts (W2, Q2). At the same probe budget the local and strided layout is what gives the map its chunk-level resolution. We compared it with an overlapping layout (local spans shifted by half a span) and with two random balanced partitions, with the same 2K probes and the same input tokens. LongBench v2 has no evidence annotations, so it cannot isolate the contribution of the layout cleanly. So we measured this on the same BABILong questions as above, where the positions of the evidence are known.
A local span scores one region of the text, so local spans alone can place the evidence among the K chunks read (top-K) but cannot say which chunk it is. A strided span takes one chunk from every region, so the two together point at a single chunk (top-1). With overlapping spans about ten chunks still tie at the peak, so the map names a region and not a chunk. Random partitions separate the chunks as well, but none of their spans is contiguous text, and they localize less well on both measures. The selected read needs the evidence among its K chunks and not at the peak, so its accuracy moves with top-K, and less than top-1 does.
Aggregation rule (W2, Q2). The maximum is above the sum and the soft maximum with all three backbones. The table recomputes the direct readout on LongBench v2 from the same probes under each rule.
A sum adds the scores of the spans that hold no evidence, and the maximum does not.
Retrieval and agentic baselines (W1, W2, Q3). We ran BM25, embedding retrieval, a trained reranker over every chunk and the reading agent ReadAgent, next to Chain of Agents. Each selector reads the same K chunks as Periscope select, and ReadAgent may reread up to K pages. The table gives each method on three axes, on LongBench v2 with Qwen3.5-4B: accuracy, compute (PFLOPs per question, as in Table 1) and memory (the largest call of a question, which the KV cache must hold).
No method in the table is more accurate than Periscope select, and only the window read is level with it, with a call 9 times longer. On Llama-3.1-8B (InfiniteBench En.MC) we also ran two training-free long-context methods with their official code: ParallelComp scores 66.8% and EM-LLM 44.8%, against 76.9% for Periscope select.