acceptodds
Under review as a conference paper at ICLR 2027

Compute-Optimal Agentic RAG

Abstract

Long-context language models can read whole documents, but paying for every token on every question is costly. An agent can instead bring document information into its context through a question-independent summary, written once and reused, or through retrieval at question time. How should inference compute be split between summary length, retrieval and model size? We vary five levers—summary length, number of retrieval calls, passages per call, passage length and model size (Qwen3.5, 0.8B to 27B parameters)—on LongBench-v2, over about 860k question–configuration pairs, and count each question's theoretical FLOPs exactly. Summaries and retrieval supply one resource, document information, and substitute for each other: the longer the summary, the less retrieval pays. Model size is a separate axis, with a threshold near 4B parameters below which the levers help little. A 15-parameter law built on this structure predicts lever combinations it was not fit on close to the limit set by sampling noise. Its compute-optimal allocation is simple: past the threshold, and for documents queried often enough to amortize the summary, spend about half of each question's compute on the summary and keep retrieval to a few thousand tokens; for a document queried once, retrieval and a larger model are best. An agent left to decide how often to search stays about 7 accuracy points below this allocation.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
long-context language models, inference-compute allocation, retrieval-augmented generation, document summarization, inference scaling laws, cost-accuracy trade-offs, theoretical FLOPs, LongBench-v2, Qwen3.5, model capability threshold

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1KJsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how inference compute should be allocated across several 'levers' in long-document agentic RAG. The five levers are summary length, number of retrieval calls, passages per call , passage length, and model size (Qwen3.5 0.8B–27B). On LongBench-v2, it runs roughly 860k question–configuration pairs, with compute precisely charged in theoretical FLOPs.

Critical strengths

1.The experimental scale and engineering effort are genuinely substantial. A cross-product grid of 860,762 question–configuration pairs, five model sizes, and five levers is not a dataset most labs could assemble. 2.The compute accounting is the highlight part of this paper. The per-token FLOPs decomposition (dense / attention / head) surfaces an easily overlooked detail—at 0.8B, the output projection accounts for 49% of the dense per-token cost—and it is aligned item by item against a reference implementation using PyTorch's FLOP counter; the append-only trajectories are checked and the 2 out of 860,764 violating units are removed. Counting generated tokens (which Yue et al. 2024 treat as negligible) is the correct call in this setting. 3.The methodological honesty is rare. Table 9 explicitly lays out the temporal order of “fit → record predictions → wait for runs to finish → score,” the selection rule for the four candidate retrieval terms is fixed before they are compared, and Test 1 / Test 2 are entirely prediction without refitting. This is far more credible than the vast majority of work that claims predictive capability. 4.There is a clear awareness of the paper's own evaluation traps. It explicitly distinguishes the “empirical frontier” (subject to selection bias, +3 points) from the “law's optimal path,” and demonstrates that bias with a half-sample split (−0.5 ± 1.1 points). 5. The outputs are actionable. The compact recipe, the allocation under constraints (fixing two passages per call costs a median of only 0.7 points), and the analysis amortizing summary cost over Q queries (Fig. 4b / Fig. 13) are directly useful for engineering practice. 6. The limitations and the AI use statement are written in a restrained and specific way, without embellishment.

Critical weaknesses

1.Novelty and theoretical depth are moderate. The substitutability between summarization and retrieval is a natural corollary of CES aggregation, and the main text itself states that the model-free test, not the fitted 'RHO', is the evidence for substitutability. The so-called 'law' is a curve fit of 15 free parameters to 2,456 configurations, with no derivation, and the authors admit that the parameters hold only for this setting. It is closer to a highly compressed empirical summary than to a transferable scaling law. I ask the authors to answer explicitly: beyond this setting, what structural, non-trivial prior can readers take away? 2.Limited generalizability is a serious weakness. A single model family, a single benchmark, a single retriever, and a single summarizer. A median noise level of 2.4 points means that many key differences sit at the edge of noise. I ask the authors to answer explicitly: do you make experiment at any other model? or how you ensure your conclusion is valid to other models?

Questions for authors

Same as Weaknesses

1 comment
AuthorsOpenReview ↗

Response to reviewer 1KJs

Short answers. (1) Is substitutability only a consequence of the CES form? Yes as a mathematical matter, so we do not use the fitted exponent as evidence; the evidence is a model-free comparison, described below. (2) Other models: not yet; what we have tested, and how we plan to test other families, is below.

Novelty, the CES form, and what carries over (W1). You are right that a CES aggregate can represent substitution, and that a fit of its exponent does not by itself show that summaries and retrieval substitute. That is why the paper rests the claim on a test that does not use the equation. At 9B, a doubling of the passages per call is worth +2.8 points of accuracy without a summary and +0.8 with a long one, and the same holds for the number of calls and passage length; a long summary's advantage shrinks from +14 to +3 points as retrieval grows but never becomes negative. So a summary replaces retrieval rather than making it more productive. What the equation adds is the strength of the substitution (the fitted exponent 0.66 corresponds to an elasticity of about 3: substantial but not perfect) and a way to turn it into allocation numbers. We agree the law is an empirical summary, not a derivation. What we think a reader can take beyond this setting, with evidence (Comments 1 and 2):

  1. The form is reusable. With three numbers calibrated per benchmark, the same 15 shared parameters fit LongBench-v2 (2.36) and LooGLE (2.71); the additive-in-log alternative is worse on both (2.51 and 3.70). Fitting both benchmarks together changes the shared exponents (substitution, distinct passages, passage length) and size slopes by at most 8%.
  2. What differs between benchmarks is small and identifiable: a difficulty shift, the summary weight (0.30 on LongBench-v2, 0.12 on LooGLE) and the guessing floor.
  3. How little calibration is needed: 12-16 configurations drawn at random from the new benchmark's grid give an error of 3.4-3.6 points (post hoc), against 1.9 when everything is refitted.
  4. Summaries are a lever worth modelling: without one, decisions lose about 4 points at equal cost. We do not claim a universal law: used as is, the law was 16 points off on LooGLE.

Other models, and noise (W2). Other models: we have not yet run another model family. Our experiments cover five Qwen3.5 sizes (0.8B to 27B) and two benchmarks, so we limit our claims to what we tested: the paper's conclusions hold for Qwen3.5, and what we claim about the equation is its form, not its parameters. We plan to test Gemma 4 and Nemotron 3.5, with the predictions fixed before the runs, and will report the results either way. Noise: each configuration is run on a finite set of questions, so its measured accuracy is uncertain by about 2.4 points; part of that is the agent's own randomness, since repeating the same runs changes a configuration's accuracy by about 1.3 points. Comparing two configurations on the same questions cancels some of the question-to-question variation but not all of it: a typical difference between two configurations is uncertain by about 2.7 points (a 95% interval of about +-5). So we agree that a single difference of a few points between two nearby configurations is within noise. Most effects in the paper are averages or slopes over many configurations (for example, the gain per doubling of the passages per call), which reduces this noise, and the equation's error on held-out configurations (2.36 points) is about the noise level. The revision will give an interval for each reported effect and mark the differences that are not resolved.

Single retriever and single summarizer. Summarizer: after submission we compared whole-document summaries with section-by-section summaries at matched realized length and found similar accuracy (within about 3 points) on QuALITY up to 67% retention, on LongBench v1, and on LongBench-v2 at low retention. Retriever: dense-only, BM25-only and hybrid retrieval agree within about one point on the same 9B retrieval cells (details in Comment 5), so the allocation conclusions do not depend on the dense component here; this is one agent size and two passage lengths.

Reviewer_ZXJvOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies how to make long-document question answering more accurate and compute-efficient.

The authors experiment with different Qwen3.5 model sizes, summary lengths, and retrieval settings to understand how they affect accuracy and compute cost.

They find three main things:

  1. Model capability matters: Smaller models benefit less from additional information.
  2. Summarization and retrieval can substitute for each other: Longer summaries reduce the benefit of additional retrieval.
  3. The best strategy depends on usage: Retrieval is generally better for one-time questions, while reusable summaries become more efficient when documents are queried repeatedly.

Finally, the authors propose a mathematical model to predict performance and identify compute-efficient configurations.

Critical strengths

1. Important and practically relevant research problem.

The paper addresses a meaningful and practical problem: how to efficiently allocate limited inference compute for long-document question answering. Finding the right balance between model size, summarization, and retrieval is important for building efficient RAG systems.

2. Extensive and systematic experiments.

The paper conducts a large-scale experimental study with approximately 860,000 question–configuration evaluations. It systematically explores different model sizes, summary lengths, and retrieval settings, providing valuable empirical insights into how these factors affect accuracy and computational efficiency.

3. Quantitative analysis and predictive modeling.

Beyond reporting experimental results, the authors propose a mathematical model to predict performance under different configurations. The evaluation on unseen configurations provides additional evidence supporting the effectiveness of the proposed model.

Critical weaknesses

1. Lack of clarity and readability.

Although the research question is introduced, the paper does not explain its motivation and main contributions in a sufficiently clear and intuitive way. The extensive experiments and mathematical formulations make the paper difficult to follow. A clearer explanation of the problem, the proposed approach, and the key takeaways would significantly improve readability.

2. Unclear novelty and insufficient comparison with prior work.

Several prior studies have already investigated inference compute allocation and scaling laws for RAG, such as Yue et al. (ICLR 2025), Zhou et al. (AAAI 2026), and Li et al. (EMNLP 2024). Although this paper considers reusable summaries and agentic retrieval, its additional contributions over existing research are not sufficiently established.

In particular, Zhou et al. (AAAI 2026) have already proposed a joint inference scaling law for RAG, but this closely related work does not appear to be cited or discussed. The authors should better clarify the novelty of their approach and provide stronger comparisons with relevant baselines.

3. Limited generalizability.

All experiments are conducted using a single model family (Qwen3.5) and a single benchmark (LongBench-v2). This raises concerns about whether the findings are specific to Qwen rather than general properties of agentic RAG. Experiments with other model families and datasets are needed to support the generalizability of the conclusions.

4. Incomplete experimental and reproducibility details.

Although the paper provides some model and inference settings, important information about the experimental infrastructure is missing, such as GPU types, GPU counts, and total experiment runtime. Given the large experimental scale, these details are important for understanding the computational requirements and reproducibility of the study.

5. Limited evaluation of real-world computational efficiency.

The paper primarily uses theoretical FLOPs to evaluate computational efficiency. However, real-world deployment also involves inference latency, memory usage, retrieval overhead, and infrastructure costs. It remains unclear whether the proposed compute-optimal configurations would remain optimal under practical deployment conditions.

References

  1. Yue et al. (2025). Inference Scaling for Long-Context Retrieval Augmented Generation. ICLR 2025.

  2. Zhou et al. (2026). Inference Scaling Law for Retrieval Augmented Generation. AAAI 2026.

  3. Li et al. (2024). Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach. EMNLP 2024, Industry Track.

Questions for authors
  1. What are the main differences between this work and prior studies, such as Yue et al. (2025) and Zhou et al. (2026)?

  2. Can the authors provide experiments on other model families and datasets to support their conclusions?

  3. Could the authors provide more details about the hardware setup, GPU usage, and total experiment runtime?

  4. Would the proposed compute-optimal configurations still be optimal when considering real-world costs, such as inference latency and retrieval overhead?

3 comments
AuthorsOpenReview ↗

Response to reviewer ZXJv 1/2

Short answers. (1) Zhou et al. (AAAI 2026) is a paper we missed; the differences are below and in Comment to all reviewers 1/2. (2) We added a second benchmark (Comment to all reviewers 2/2); other model families are planned. (3) Hardware is given below. (4) Real-world cost (Q4): the next comment.

How this differs from Zhou et al. (W2, Q1). Zhou et al. ("Inference Scaling Law for Retrieval Augmented Generation", AAAI 2026) fits F1 against four factors (retriever size, generator size, number of retrieved documents, context window) with power-law and sigmoid curves and a joint form, for zero-shot T5 and OPT models on NQ, TriviaQA, WebQuestions and FEVER, refitting parameters for each dataset. It reports in-sample fit (R^2) and we found no held-out prediction in it. It allocates compute between retriever and generator with a latency model. It has no summary lever, no agent that decides how many retrieval calls to make, no discount for repeated passages, and no cost that includes generated tokens (Comment to all reviewers 1/2). We will cite and discuss it; Li et al. (EMNLP 2024 Industry) is already cited.

Other models and datasets (W3, Q2). A second benchmark is in Comment to all reviewers 2/2. Other model families are not yet tested; we plan Gemma 4 and Nemotron 3.5, each with its own FLOPs accounting and predictions frozen before the runs, and will not claim transfer across families until then.

Hardware (W4, Q3). All runs used NVIDIA A100-SXM4-80GB GPUs. The evaluation and summarization jobs of the study took about 10,000 GPU-hours (job duration times GPUs); this includes time lost to preemptions and restarts and some failed or smoke runs, so it is an upper bound on busy GPU time. We will add this to the appendix.

Real-world cost (W5, Q4). Answered in the next comment.

Clarity (W1). Presentation scores range from 1 to 3 across the three reviews, and we take the point. The revision will open with the question, the three findings and a one-paragraph recipe, and move the model formulation after them.


Real-world cost (W5, Q4)

The question (Q4): would the compute-optimal configurations still be optimal when real-world costs, such as inference latency and retrieval overhead, are taken into account?

Answer in short.

  • Latency: largely yes. If we choose configurations by FLOPs but the real cost is closer to latency, we lose about 1 point of accuracy on average, compared with choosing by latency. The recommended pattern (a summary plus about two retrieval calls) stays the same.
  • Retrieval overhead: it depends on how often a document is queried. If each document is queried once, as in LongBench-v2, adding the cost of embedding the document makes retrieval configurations about 68% more expensive and changes which configurations are best. If a document is queried ten times, the extra cost is 7% and little changes. The overhead can also be avoided: BM25-only retrieval needs no embedding and was as accurate as dense or hybrid retrieval on this benchmark.
  • The revision will present allocations as a function of these two costs, so that practitioners can read off the recommendation for their own serving setup.

Details follow in 2/2.

AuthorsOpenReview ↗

Response to reviewer ZXJv 2/2: real-world cost (W5, Q4)

A. Latency

The issue. Our cost counts FLOPs, so one generated token counts the same as one token the model reads (a prompt token). On a GPU, generating a token is much slower than reading one, because generation is limited by memory bandwidth and reading is limited by arithmetic. How much slower depends on how many requests the server handles at the same time (the batch size).

What we measured. On one A100-80GB with vLLM (9B model; 2B and 27B give similar values), we timed prefill (reading) and decoding (generating). We call the result the generation cost factor: for the same number of FLOPs, generating takes this many times longer than reading. Our FLOP cost corresponds to a factor of 1 (a FLOP of generating costs the same as a FLOP of reading).

requests served together (batch size)generation cost factor
1114-140
429-38
168-15
643-6

What we did with it. For each of the 2,500 LongBench-v2 configurations we recomputed the cost as: cost of reading, plus the factor times the cost of generating. We tried factors of 3.5, 10, 33 and 125, which correspond to the four batch sizes above. For each factor we then asked two questions:

  1. Of the 25 configurations that are optimal by FLOPs (no other configuration is cheaper and more accurate), how many are still optimal when the cost is recomputed with this factor?
  2. If a practitioner followed our FLOP-based recommendation although the real cost is the recomputed one, how much accuracy do they lose? For each of the 25 recommended configurations we find the most accurate configuration that costs no more, in recomputed cost, and take the difference in accuracy; the table gives the average over the 25 and the largest.
generation cost factor (batch size)FLOP-optimal configurations still optimal (of 25)accuracy lost by following the FLOP-based recommendation, average (largest), in points
3.5 (64)210.3 (3.9)
10 (16)170.8 (4.2)
33 (4)170.8 (4.2)
125 (1)161.05 (5.3)

Takeaway. The set of best configurations does change (4 to 9 of the 25 are replaced), but the replacements are close in accuracy: following the FLOP-based recommendation loses about 1 point on average even when generation is 125 times as costly. The recommended pattern stays: a summary plus about two retrieval calls. Configurations with a summary become somewhat less common among the optimal ones (84% at factor 1, 75-77% above).

A related measurement. Reusing a cached document prefix is cheap: a GPU prefix-cache hit takes 0.6-3% of the time of the cold request at 32k-131k tokens (9B: 2.7 s down to 0.07 s at 32k tokens), which favours summaries and cached documents that are read many times.

B. Retrieval overhead

The issue. Dense retrieval needs an embedding of every chunk of the document, computed once per document. Our FLOP cost counts the model's reading and generating but not this embedding.

How large it is. Embedding a document once with BGE-large costs a median of about 65-70 TFLOPs per document (13 for a short one at the 10th percentile, about 210 for a long one at the 90th). A retrieval configuration costs a median of about 110 TFLOPs per question, so the overhead is comparable to a question. It is shared by all questions asked about the same document, as the cost of a summary is.

Effect on the recommendation. We added the embedding cost, divided by the number of questions asked per document, to every retrieval configuration, and asked how many of the 25 FLOP-optimal configurations stay optimal.

questions asked per documentextra cost of the median retrieval configurationFLOP-optimal configurations still optimal (of 25)
1 (LongBench-v2)+68%15
3+23%16
10+7%19
30+2%22

Takeaway. Retrieval overhead matters when a document is queried once or twice, and little when it is queried ten times or more. Retrieval configurations still make up 86% of the optimal ones at one question per document.

C. Can the overhead be avoided?

On the same retrieval cells (9B, 402 questions, 2 or 8 calls, 1 or 4 passages, chunks of 128 and 512 tokens), dense-only, BM25-only and hybrid retrieval (what the paper uses) agree within about one point:

passage length (tokens)dense onlyBM25 onlyhybrid
12838.938.338.4
51240.540.941.6

(mean accuracy, %; differences between any two have 95% intervals of about +-2). BM25 needs no embedding, so on this benchmark the indexing cost can be avoided with no measurable loss of accuracy. This is one agent size and we do not claim it for other benchmarks.

thank you for your response, I have updated the score.

Reviewer_TbDSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate inference compute between summaries, retrieval, and model size for long-document question answering. The authors evaluate five Qwen3.5 model sizes on LongBench-v2 while varying summary length, retrieval calls, passages per call, and passage length. They propose a 15 parameters empirical model to predict accuracy and select configurations under compute budgets. They validate its predictions on new configurations and examine how document length and summary reuse affect the allocation.

Critical strengths
  1. The paper quantifies effects that are often discussed only qualitatively. Several findings may seem intuitive, such as longer summaries reducing the need for retrieval and repeated retrieval providing less new information. The value of this work is in measuring these effects, identifying when they matter, and showing how they change compute allocation. This provides more useful guidance than the general advice to use more context or a larger model.

  2. The prediction tests strengthen the empirical results. The authors record predictions before later runs finish and examine where those predictions fail. They then account for repeated retrieval and test the revised model on further configurations. This process makes the analysis more convincing than reporting a good fit to the data used to build the model.

Critical weaknesses
  1. The added value of the specific equation is not fully clear. Earlier work, including Yue et al. (2024), already models RAG performance to guide compute allocation. This paper extends that setting in useful ways, but the flexible model in Table 8 achieves similar prediction accuracy. A comparison of the configurations selected by these models would help show whether the proposed structure improves practical decisions beyond providing a compact description of the results.

  2. The allocation rules have limited external validation. All experiments use one model family, one benchmark, and one retrieval and summarization pipeline. The prediction tests cover new configurations of model sizes that have already been calibrated. They do not establish transfer to unseen models or datasets. The authors acknowledge this limitation, but it remains unclear how much of the proposed structure would hold in another setting without substantial refitting.

Questions for authors
  1. In each holdout evaluation in Table 8, was the distinct-passage predictor also trained without the held-out configurations?

  2. At the same FLOP budget, how do configurations selected by the proposed model compare with those selected by the flexible model on held-out documents?

  3. On another model family or benchmark, can the shared structural parameters remain fixed while only the model-specific terms are refitted? A small targeted evaluation would help distinguish general relationships from effects specific to the current setup.

2 comments
AuthorsOpenReview ↗

Response to reviewer TbDS

Short answers. (Q1) Yes, the distinct-passage predictor was refit without the held-out configurations. (Q2) The equation and the flexible surface choose equally good configurations; the equation's added value is the summary lever, compactness and transfer (Comment to all reviewers 1/2). (Q3) On a second benchmark the shared parameters stay fixed and only three numbers are calibrated (Comment to all reviewers 2/2).

Q1. In every held-out evaluation of Table 8 the distinct-passage predictor was refit on the training configurations only; for the test on later configurations, it was fit on the data available when the predictions were recorded.

What does the specific equation add (W1, Q2)? The comparison you suggested is the table in Comment to all reviewers 1/2: models refit on four fifths of the documents choose configurations at 12 FLOP budgets, and the choices are scored on the held-out fifth. The law and the flexible surface choose equally good configurations (45.2% and 45.4%) and the same one in 70% of cases. The additive form with a summary lever is not distinguishable either; a model that cannot use a summary loses about 4 points. So the equation does not buy better decisions than the flexible surface, and its value lies in the summary lever, a compact interpretable form, and transfer.

Transfer without substantial refitting (W2, Q3). Comment to all reviewers 2/2 is this test: the 15 shared parameters stay fixed and only a shift, the summary weight and the floor are calibrated per benchmark. On LooGLE this gives an error of 3.4-3.6 points with 12-16 randomly drawn configurations, and the flexible surface calibrated the same way is slightly worse. Other model families are untested; Comment to all reviewers 2/2 gives the plan.

Thanks for the additional experiments. The LooGLE results partly address my concern, though the stronger transfer results rely on post hoc calibration and other model families remain untested. Configuration selection is comparable to the alternatives. I maintain my rating.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Comment to all reviewers (2/2): a second benchmark, LooGLE

Question: if the equation fitted on LongBench-v2 is used on a different benchmark, what carries over and what must be re-estimated? LooGLE has long-document questions with free-form answers. We used the same five Qwen3.5 sizes and about 400 retrieval and summary configurations, each run on the same 1,101 questions. Before running the test configurations we froze the predictions, the calibration protocol and the metrics. Errors are root-mean-square error in accuracy points. We distinguish two things:

  • Calibrating: keep the 15 parameters fitted on LongBench-v2 and fit only three numbers for the new benchmark (a difficulty shift, the weight given to summaries against retrieval, and the guessing floor) on a few configurations of that benchmark.
  • Refitting: re-estimate all 15 parameters from scratch on the new benchmark, which needs far more data.

What we found, and what it means:

  1. Absolute accuracy does not transfer without calibration. Used as is, the law is 16.1 points off, because LooGLE is a different difficulty. Adjusting only the difficulty offset so that the law matches the model's accuracy with no document at all ("closed book") gives 7.4. Calibrating the three numbers on 25 configurations chosen in advance (five per size: closed book, summary only at two lengths, retrieval only, and one configuration combining a summary with retrieval) gave 8.4. So a practitioner needs calibration, and, as the next point shows, which configurations are used to calibrate matters more than how many.
  2. The form carries over, if the calibration configurations resemble the ones to be predicted. After seeing that result we repeated the calibration with configurations drawn at random from the benchmark's own grid, most of which combine a summary with retrieval, like the configurations to be predicted; this part is post hoc. Calibrating the three numbers on 12-16 such configurations (fewer than the 25 above) predicts the other configurations with an error of 3.4-3.6 points, and on 25 of them 3.2; calibrating on all the others (cross-validated) gives 3.0. Refitting all 15 parameters gives 1.9 but needs all the data. LooGLE's own noise level is about 0.9-1.5 points. So the shared structure can stay fixed while only a few numbers are fitted, at an error of about 2-3 times the noise, provided the calibration includes configurations that use both a summary and retrieval. The first set was a poor choice: it contained mostly closed-book, summary-only and retrieval-only configurations, which say little about how much weight a summary should get.
  3. What differs between benchmarks is small. Fitting both benchmarks together, with three numbers per benchmark and the other 15 parameters shared, leaves LongBench-v2 at 2.36 (2.35 when fitted on LongBench-v2 alone) and moves the shared exponents and slopes by at most 8%. The summary weight differs: 0.30 on LongBench-v2 and 0.12 on LooGLE, so summaries matter less for LooGLE's questions and the best configuration differs between benchmarks.
  4. Our equation transfers better than the alternatives. Under the same joint fit, the additive-in-log adaptation is worse on both benchmarks (2.51 against 2.36 on LongBench-v2; 3.70 against 2.71 on LooGLE). Calibrating the flexible surface the same way (a shift, the floor, and a multiplier on its summary terms) is slightly worse than the law with few cells (error 3.8 for the flexible surface against 3.3 for the law on 16 random configurations; 8.7 against 6.4 on full-context configurations, which lie outside the fitted range of summaries), and better only when all its parameters are refitted (1.6 against 1.9).

What the revision will add. (a) Other model families: Gemma 4 and Nemotron 3.5, each with its own FLOPs accounting, with predictions frozen before the runs. (b) A fresh prospective test on NarrativeQA with the calibration budget (20 configurations) fixed in advance, then InfiniteBench and NovelHopQA. (c) The cost model extended to latency and indexing (Comment 5). (d) A restructured introduction. None of this is ready within this period, and we do not rely on it here; we have not tested other model families and make no claim for them.

AuthorsOpenReview ↗

Comment to all reviewers (1/2): how our work differs from prior work, and what our equation adds

We thank the reviewers. Their questions fall into four groups: (1) how this work differs from prior scaling and allocation models and what our equation adds (this comment); (2) evidence beyond one benchmark and one model family (Comment to all reviewers 2/2); (3) real-world cost; (4) clarity. We did not change the submitted PDF. Below we report new analyses and what the revision will add; each reviewer's remaining questions are answered in their own comment.

What distinguishes our work. To our knowledge, no earlier compute-allocation or scaling model for retrieval-augmented generation includes a reusable, question-independent summary as a lever. Systems such as RAPTOR, MemWalker and MemoRAG use summaries or global memories, but do not model what a summary buys against retrieval and model size at equal compute. Yue et al. (ICLR 2025) model documents, in-context examples and iterations for a single model, and exclude generated tokens from cost; Zhou et al. (AAAI 2026) model retriever size, generator size, number of documents and context window with per-dataset parameters. We model the summary, several agent-issued retrieval calls (with a discount for repeated passages) and five model sizes in one cost measure: exact FLOPs, including generated tokens, which are the largest share of compute at most points in our setting. We also freeze predictions before running configurations, and now test a second benchmark (Comment to all reviewers 2/2).

What the equation adds: the configurations each model chooses. Yue et al.'s functional form (accuracy additive in the logarithm of each lever) has no summary or model-size lever, so we applied their functional form to our levers; every "additive-in-log" number below and in the other comments uses this adaptation. We split the 469 documents into five folds. For each fold we refit every model on the other four, including the distinct-passage predictor (the small model that predicts how many distinct passages a configuration retrieves, which the equation needs). At 12 FLOP budgets (quantiles of configuration cost) each model picks the configuration with the highest predicted accuracy. The pick is made before the held-out documents are used, and scored by that configuration's accuracy on them.

picks made byaccuracy of the picks on held-out documents (%)difference from our law (95% CI)
our law (15 parameters)45.2
flexible 22-parameter surface (a regression with no structure, our reference for how well any curve can fit)45.4+0.2 (-1.2 to +1.6)
additive-in-log form of Yue et al., applied to our levers (15 parameters)44.4-0.8 (-2.5 to +1.4)
best observed configuration on the training documents (no model)37.5-7.7 (-13.3 to -2.8)
our law, but only configurations without a summary may be picked40.9-4.2 (-8.3 to +0.1)
additive form without a summary lever (as in Yue et al.), picks restricted likewise40.9-4.3 (-8.2 to -0.1)

How to read it:

  • The summary lever is what matters for decisions. A model that cannot use a summary, whatever its form, can only pick configurations without one, and loses about 4 points of accuracy at equal cost: the summary is often the best use of compute. Among configurations without a summary the form hardly matters (40.9% for both).
  • The specific equation does not choose better than the alternatives that include a summary lever. It picks the same configuration as the flexible surface in 70% of cases, and is not distinguishable from the additive form on choices. The test has limited power (five folds, 12 budgets). We will say this plainly in the revision.
  • What the equation does add is (a) a compact, interpretable description (how summaries and retrieved passages substitute for each other, a term that counts only distinct passages when calls repeat, a capability curve per model size) and (b) better prediction and transfer: on held-out configurations the law is closer than the additive form (2.36 against 2.51 points on LongBench-v2), and on a second benchmark the gap widens (Comment to all reviewers 2/2).

Related work. Zhou et al. (AAAI 2026) is a paper we missed. We will cite and discuss it; see the comment for ZXJv for how it differs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.