- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how inference compute should be allocated across several 'levers' in long-document agentic RAG. The five levers are summary length, number of retrieval calls, passages per call , passage length, and model size (Qwen3.5 0.8B–27B). On LongBench-v2, it runs roughly 860k question–configuration pairs, with compute precisely charged in theoretical FLOPs.
Critical strengths
1.The experimental scale and engineering effort are genuinely substantial. A cross-product grid of 860,762 question–configuration pairs, five model sizes, and five levers is not a dataset most labs could assemble. 2.The compute accounting is the highlight part of this paper. The per-token FLOPs decomposition (dense / attention / head) surfaces an easily overlooked detail—at 0.8B, the output projection accounts for 49% of the dense per-token cost—and it is aligned item by item against a reference implementation using PyTorch's FLOP counter; the append-only trajectories are checked and the 2 out of 860,764 violating units are removed. Counting generated tokens (which Yue et al. 2024 treat as negligible) is the correct call in this setting. 3.The methodological honesty is rare. Table 9 explicitly lays out the temporal order of “fit → record predictions → wait for runs to finish → score,” the selection rule for the four candidate retrieval terms is fixed before they are compared, and Test 1 / Test 2 are entirely prediction without refitting. This is far more credible than the vast majority of work that claims predictive capability. 4.There is a clear awareness of the paper's own evaluation traps. It explicitly distinguishes the “empirical frontier” (subject to selection bias, +3 points) from the “law's optimal path,” and demonstrates that bias with a half-sample split (−0.5 ± 1.1 points). 5. The outputs are actionable. The compact recipe, the allocation under constraints (fixing two passages per call costs a median of only 0.7 points), and the analysis amortizing summary cost over Q queries (Fig. 4b / Fig. 13) are directly useful for engineering practice. 6. The limitations and the AI use statement are written in a restrained and specific way, without embellishment.
Critical weaknesses
1.Novelty and theoretical depth are moderate. The substitutability between summarization and retrieval is a natural corollary of CES aggregation, and the main text itself states that the model-free test, not the fitted 'RHO', is the evidence for substitutability. The so-called 'law' is a curve fit of 15 free parameters to 2,456 configurations, with no derivation, and the authors admit that the parameters hold only for this setting. It is closer to a highly compressed empirical summary than to a transferable scaling law. I ask the authors to answer explicitly: beyond this setting, what structural, non-trivial prior can readers take away? 2.Limited generalizability is a serious weakness. A single model family, a single benchmark, a single retriever, and a single summarizer. A median noise level of 2.4 points means that many key differences sit at the edge of noise. I ask the authors to answer explicitly: do you make experiment at any other model? or how you ensure your conclusion is valid to other models?
Questions for authors
Same as Weaknesses
Response to reviewer 1KJs
Short answers. (1) Is substitutability only a consequence of the CES form? Yes as a mathematical matter, so we do not use the fitted exponent as evidence; the evidence is a model-free comparison, described below. (2) Other models: not yet; what we have tested, and how we plan to test other families, is below.
Novelty, the CES form, and what carries over (W1). You are right that a CES aggregate can represent substitution, and that a fit of its exponent does not by itself show that summaries and retrieval substitute. That is why the paper rests the claim on a test that does not use the equation. At 9B, a doubling of the passages per call is worth +2.8 points of accuracy without a summary and +0.8 with a long one, and the same holds for the number of calls and passage length; a long summary's advantage shrinks from +14 to +3 points as retrieval grows but never becomes negative. So a summary replaces retrieval rather than making it more productive. What the equation adds is the strength of the substitution (the fitted exponent 0.66 corresponds to an elasticity of about 3: substantial but not perfect) and a way to turn it into allocation numbers. We agree the law is an empirical summary, not a derivation. What we think a reader can take beyond this setting, with evidence (Comments 1 and 2):
Other models, and noise (W2). Other models: we have not yet run another model family. Our experiments cover five Qwen3.5 sizes (0.8B to 27B) and two benchmarks, so we limit our claims to what we tested: the paper's conclusions hold for Qwen3.5, and what we claim about the equation is its form, not its parameters. We plan to test Gemma 4 and Nemotron 3.5, with the predictions fixed before the runs, and will report the results either way. Noise: each configuration is run on a finite set of questions, so its measured accuracy is uncertain by about 2.4 points; part of that is the agent's own randomness, since repeating the same runs changes a configuration's accuracy by about 1.3 points. Comparing two configurations on the same questions cancels some of the question-to-question variation but not all of it: a typical difference between two configurations is uncertain by about 2.7 points (a 95% interval of about +-5). So we agree that a single difference of a few points between two nearby configurations is within noise. Most effects in the paper are averages or slopes over many configurations (for example, the gain per doubling of the passages per call), which reduces this noise, and the equation's error on held-out configurations (2.36 points) is about the noise level. The revision will give an interval for each reported effect and mark the differences that are not resolved.
Single retriever and single summarizer. Summarizer: after submission we compared whole-document summaries with section-by-section summaries at matched realized length and found similar accuracy (within about 3 points) on QuALITY up to 67% retention, on LongBench v1, and on LongBench-v2 at low retention. Retriever: dense-only, BM25-only and hybrid retrieval agree within about one point on the same 9B retrieval cells (details in Comment 5), so the allocation conclusions do not depend on the dense component here; this is one agent size and two passage lengths.