- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates scaling proactive compute -- computation spent by LLM agents during otherwise idle periods before a query arrives or while waiting for tool execution to finish -- to improve test-time efficiency and performance. Its main contributions are: 1) quantifying how proactive compute shifts the accuracy–test-time-compute frontier and reduces amortized token costs when studies are reused across shared contexts; 2) introducing hierarchical wiki-style and multi-agent partitioning strategies to scale proactive study beyond a single context window; and 3) explaining the underlying mechanisms behind these gains through trace analyses, content interventions, and compression experiments. It validates these findings across nine offline benchmarks (spanning math, Algorithmic, SWE, and Legal) and three online MLE-Bench tasks using multiple model families, demonstrating that proactive preparation boosts test-time efficiency by 9 to 48 percentage points, reaches stronger online scores earlier under the same wall-clock budgets, and cuts downstream reasoning tokens by more than half and tool calls roughly tenfold at the same accuracy.
Critical strengths
The paper’s primary strength is its controlled evaluation of how idle-time preparation -- both pre-query study and online tool-wait planning -- trades off against downstream reactive compute across diverse benchmarks. Rather than merely showing that extra tokens improve accuracy, it decouples proactive and test-time compute budgets and uses targeted ablations to explain when and why upfront preparation reduces downstream reasoning tokens and tool calls. This is consequential to the overall assessment because it offers actionable insights into the economics and amortization of idle-time compute, elevating the work beyond a simple prompting heuristic despite its other limitations.
Critical weaknesses
-
The presentation of the empirical results needs significant improvement to support the paper's claims clearly. For instance, in Figure 2 (as well as Figures 4, 7), up to nine overlapping curves are plotted in tiny subplots using a monochromatic green gradient with nearly indistinguishable shades, making it very difficult to tell different proactive budgets P apart or verify the exact scaling trends across intermediate budgets.
-
The proposed approach relies primarily on hand-crafted prompting heuristics rather than a principled mechanism for learning what or when to study. Furthermore, on repository and document benchmarks, the paper fails to compare against standard, lightweight non-LLM baselines (such as BM25/dense retrieval or static indexing) that load relevant files without expensive LLM study phases, making it unclear if generating up to 1M+ study tokens is practically justified.
-
Headline efficiency gains are heavily driven by artificial math splits where the "context" holds the entire problem except the final sentence, allowing the agent to virtually solve the problem before the query arrives. Meanwhile, on realistic tasks, the cost-amortization analysis counts only generated output tokens -- ignoring the substantial input/prefill cost of repeatedly reading 64k–128k study tokens across multi-turn tool calls -- and extrapolates savings to 1,024 queries from benchmarks with up to 30 questions per context, overstating the real-world economic benefits.
Questions for authors
-
How does proactive LLM study compare against lightweight retrieval or indexing baselines (e.g., BM25, dense retrieval, or static file indexing) that pre-load relevant files without generating expensive study tokens?
-
Beyond fixed budget forcing and hand-crafted prompts, can the agent adaptively decide what to study and when to stop once sufficient context is gathered?
-
How do the cost-amortization results and break-even query counts change when you include input/prefill token costs from repeatedly reading long study context across multi-turn tool calls?
-
Excluding the artificial math splits, is proactive compute still net cost-effective on repository and document benchmarks when evaluated strictly within their actual query counts rather than extrapolating to N = 1,024?
-
Could you revise Figures 2, 4, and 7 using distinct colors or line styles and provide a summary table so readers can clearly distinguish intermediate proactive budgets P?
3 comments
Rebuttal by Authors 2/2
Agents can adaptively choose their budgets (W2, Q2)
The study prompt is the same for every offline benchmark and model, and only the context changes (Appendix A; prompts in Appendix H.1), with no per-task tuning (the tool benchmarks add a one-line tool description). On the repository benchmarks the agent already chooses every file it reads. We control the budgets with budget forcing from s1 (Muennighoff et al., 2025), because it sets the raw compute on both axes precisely, which lets us measure how each budget scales. Forcing also gives finer control than effort levels. On a SWE-QA-Pro question, medium effort spent fewer reasoning tokens than low (5,387 against 8,542), so effort levels cannot set a budget.
We keep the study budget fixed so that we can measure how the gain scales with it. The agent already decides when to stop in two places: the uncapped solver stops when it answers, and online, the uncapped agent decides how long to work during each wait. To test whether the solver can decide for itself, we removed forcing in two ways. In both, the study still cuts test-time work at about the same accuracy on SWE-QA-Pro and StudyBench (the first test is in §7.1). First, we let the solver choose its own length. On SWE-QA-Pro, a 16k study still cuts its tool calls from 46.9 to 4.5 per task at the same accuracy (Figure 8 right), and across SWE-QA-Pro, StudyBench and Harvey LAB the unforced point lies within 0.08 of the forced curve at all 24 budgets (P = 0 to 64k) we measured (mean absolute gap 0.04; a new analysis for this response). Second, we set compute only with the model’s reasoning effort. At effort off and low, as with the uncapped runs, a study still cuts tool calls at about the same accuracy (SWE-QA-Pro, effort off: 69.3 to 7.4 per task; new runs for this response). So the study still cuts test-time work without per-task prompt tuning and without forcing.
Efficiency gains hold on all benchmarks (W3, Q4)
We use the math splits as a controlled example, following Sleep-time Compute (Lin et al., 2025), in which the context nearly determines the task. Furthermore, our headline results hold on the other benchmarks too, covering 14 unique repositories on SWE-QA-Pro, 2 code repositories on StudyBench, and a 134M-token legal filesystem from Harvey LAB. On these benchmarks, the task is less predictable, so the study explores broadly and records file locations and observations instead of an answer (§4, Example 2), and the gains still hold. For example, with a 128k study, the solver at T=64 matches a no-study solver with a 15x larger budget on SWE-QA-Pro and 7x on StudyBench (§4.1; Figure 1 for SWE-QA-Pro). On Harvey LAB, E rises from 0.319 without a study to 0.589 with a hierarchical summary (§6; Table 11). The revised introduction leads with these results, and Figure 1 now shows SWE-QA-Pro.
Clarifying Figures 2, 4 and 7 (W1, Q5)
Thank you for your suggestion. We have revised these figures since submission. Figure 2 is now Figure 3. Figure 4 (Figure 5 in the revision) now shows, at 32 seeds per arm, the share of sessions that beat the median human over time, with the reactive agent drawn on top so that every budget reads against it. In Figure 7 (now Figure 8) the left panels show 4 of the 8 study budgets, each with its own marker, and the confidence bands moved to Figure 45 (Appendix G). All figures now use one uniform text size. For a summary per budget, Figure 3 (right; with 95% CIs in Figure 40, Appendix G) gives the efficiency gain for every P, and the appendix gives accuracy over the full P × T grid for each model (Appendix C.8, Figures 16, 20, 24 and 28).
We hope the new measurements address your concerns about cost and baselines, and would be grateful if you would consider raising your score.
Thanks for the detailed response! My concerns have been addressed and I have raised my rating accordingly.
Rebuttal by Authors 1/2
Thank you for the review, and for noting that decoupling the proactive and test-time budgets offers actionable insights into the economics and amortization of idle-time compute. Following your questions, we measured cost with input tokens at the real task counts and added retrieval baselines.
Input cost and query counts. Studies break even after about 2 to 10 tasks (W3, Q3, Q4)
We amortize the study over the tasks that share its context. A company repository or document store can serve thousands of tasks over weeks and months of use, so we extrapolate to 1,024 tasks. Even without that extrapolation, we find end-to-end savings in cost on just the 120 tasks of Harvey LAB (table below; Table 4, Appendix C.5). Harvey LAB is our most realistic benchmark. The savings hold across the study methods below:
The answer price is the solver’s cost for one task. Token columns are the solver’s tokens over all 120 tasks. The total price adds the study once, and the break-even point is the study price divided by the saving per task.
Prices are StreamLake’s list prices on OpenRouter for the model we ran (DeepSeek-V4-Flash-0731): 0.132 USD per million new input tokens, 0.0042 USD per million cached input tokens and 0.396 USD per million output tokens (Appendix C.5).
Every plain and wiki study up to 128k, the 262k wiki study and every parallel or serial study with up to four agents has a lower total cost than no study at the benchmark’s own count of 120 tasks at a 60% score (14 settings; the 95% bootstrap CI over tasks excludes 0 in each; Appendix C.5), against a no-study cost of $4.23. The savings grow with every further task that shares the context.
Retrieval and studying work in different regimes (W2, Q1)
Our focus is how proactive compute scales, which is orthogonal to how retrieval-augmented generation (RAG) performs. Retrieval runs after the task arrives and requires information about the task, while a study runs before it and is inherently task-blind. On AIME 2025 and ARC-AGI the solver already sees the whole source, so retrieval has nothing to add, and a 16k study still raises accuracy from 0.00 to 0.56 (AIME, T=64) and from 0.26 to 0.55 (ARC-AGI, T=4k; Appendix C.6, Figure 3).
Despite these deep differences between the two paradigms, we provide the comparison of a task-blind study with RAG in the following table (Table 5, Appendix C.6; Figure 3). BM25 and dense retrieval see the task and fill the prompt with the most relevant chunks, up to the number of prompt tokens the 16k study gives the solver (about 85k tokens). The file index is task-blind and generates no study tokens.
SWE-QA-Pro, 16k study. Study and no study: 30 tasks, 2 seeds. BM25, dense and file index: up to 24 tasks, 1 seed. * On AIME and ARC-AGI the solver already sees the whole context, so retrieval adds nothing, and we show the no-study score.
Paired on the same tasks, the task-blind study is never more than 0.01 below task-aware retrieval, and it beats the file index at T ≤ 1k. It also leaves less work for test time. At T=1k the solver makes 2.7 tool calls per task with the study, against 5.2 with BM25 and 10.4 with no study. We include this comparison for completeness, since the question of the paper is how the gain scales with proactive compute.