acceptodds
Under review as a conference paper at ICLR 2027

Scaling Proactive Compute: When the Agent Anticipates the Task

Abstract

Language model agents benefit from additional test-time compute, yet often remain idle before queries arrive and while tools run. We study how scaling *proactive compute* during these periods improves performance, with three main findings. First, controlled sweeps of preparation and downstream reasoning budgets across nine offline benchmarks show that preparation improves the accuracy–test-time-compute frontier. Reusing studies across questions can also reduce total generated-token costs. In three machine-learning engineering tasks, agents that reason while tools run reach better scores earlier under equal wall-clock budgets. Second, our wiki-style and multi-agent study strategies improve efficiency on Harvey LAB's 134M-token document collection and extend preparation beyond a single context window. Third, our analyses show how preparation reduces downstream reasoning and tool use. On SWE-QA-Pro, a 16k-token study more than halves subsequent reasoning and cuts tool calls roughly tenfold at the same accuracy, even when the solver chooses its own budget. Together, these findings establish proactive compute as an effective complement to test-time scaling and show how agents can turn otherwise idle time into useful preparation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
offline compute, amortization, test-time scaling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_q9ixOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates scaling proactive compute -- computation spent by LLM agents during otherwise idle periods before a query arrives or while waiting for tool execution to finish -- to improve test-time efficiency and performance. Its main contributions are: 1) quantifying how proactive compute shifts the accuracy–test-time-compute frontier and reduces amortized token costs when studies are reused across shared contexts; 2) introducing hierarchical wiki-style and multi-agent partitioning strategies to scale proactive study beyond a single context window; and 3) explaining the underlying mechanisms behind these gains through trace analyses, content interventions, and compression experiments. It validates these findings across nine offline benchmarks (spanning math, Algorithmic, SWE, and Legal) and three online MLE-Bench tasks using multiple model families, demonstrating that proactive preparation boosts test-time efficiency by 9 to 48 percentage points, reaches stronger online scores earlier under the same wall-clock budgets, and cuts downstream reasoning tokens by more than half and tool calls roughly tenfold at the same accuracy.

Critical strengths

The paper’s primary strength is its controlled evaluation of how idle-time preparation -- both pre-query study and online tool-wait planning -- trades off against downstream reactive compute across diverse benchmarks. Rather than merely showing that extra tokens improve accuracy, it decouples proactive and test-time compute budgets and uses targeted ablations to explain when and why upfront preparation reduces downstream reasoning tokens and tool calls. This is consequential to the overall assessment because it offers actionable insights into the economics and amortization of idle-time compute, elevating the work beyond a simple prompting heuristic despite its other limitations.

Critical weaknesses
  • The presentation of the empirical results needs significant improvement to support the paper's claims clearly. For instance, in Figure 2 (as well as Figures 4, 7), up to nine overlapping curves are plotted in tiny subplots using a monochromatic green gradient with nearly indistinguishable shades, making it very difficult to tell different proactive budgets P apart or verify the exact scaling trends across intermediate budgets.

  • The proposed approach relies primarily on hand-crafted prompting heuristics rather than a principled mechanism for learning what or when to study. Furthermore, on repository and document benchmarks, the paper fails to compare against standard, lightweight non-LLM baselines (such as BM25/dense retrieval or static indexing) that load relevant files without expensive LLM study phases, making it unclear if generating up to 1M+ study tokens is practically justified.

  • Headline efficiency gains are heavily driven by artificial math splits where the "context" holds the entire problem except the final sentence, allowing the agent to virtually solve the problem before the query arrives. Meanwhile, on realistic tasks, the cost-amortization analysis counts only generated output tokens -- ignoring the substantial input/prefill cost of repeatedly reading 64k–128k study tokens across multi-turn tool calls -- and extrapolates savings to 1,024 queries from benchmarks with up to 30 questions per context, overstating the real-world economic benefits.

Questions for authors
  • How does proactive LLM study compare against lightweight retrieval or indexing baselines (e.g., BM25, dense retrieval, or static file indexing) that pre-load relevant files without generating expensive study tokens?

  • Beyond fixed budget forcing and hand-crafted prompts, can the agent adaptively decide what to study and when to stop once sufficient context is gathered?

  • How do the cost-amortization results and break-even query counts change when you include input/prefill token costs from repeatedly reading long study context across multi-turn tool calls?

  • Excluding the artificial math splits, is proactive compute still net cost-effective on repository and document benchmarks when evaluated strictly within their actual query counts rather than extrapolating to N = 1,024?

  • Could you revise Figures 2, 4, and 7 using distinct colors or line styles and provide a summary table so readers can clearly distinguish intermediate proactive budgets P?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

Thank you for the review, and for noting that decoupling the proactive and test-time budgets offers actionable insights into the economics and amortization of idle-time compute. Following your questions, we measured cost with input tokens at the real task counts and added retrieval baselines.

Input cost and query counts. Studies break even after about 2 to 10 tasks (W3, Q3, Q4)

We amortize the study over the tasks that share its context. A company repository or document store can serve thousands of tasks over weeks and months of use, so we extrapolate to 1,024 tasks. Even without that extrapolation, we find end-to-end savings in cost on just the 120 tasks of Harvey LAB (table below; Table 4, Appendix C.5). Harvey LAB is our most realistic benchmark. The savings hold across the study methods below:

Harvey LAB, 120 tasksStudy priceAnswer price per question (excludes study)Avg output tokens per questionInput tokensCached input tokensOutput tokensTotal priceChange vs no studyTasks to break even on study cost
No study$0$0.035313.7k24.6M78.9M1.65M$4.23––
Plain study, 16k$0.027$0.020210.7k11.3M102.0M1.29M$2.45−$1.78 (−42%)1.8
Wiki study, 32k$0.053$0.01348.4k5.5M114.0M1.01M$1.66−$2.57 (−61%)2.4
Parallel agents, N=1 (128k)$0.230$0.01307.1k8.2M31.7M0.86M$1.79−$2.44 (−58%)10.3

The answer price is the solver’s cost for one task. Token columns are the solver’s tokens over all 120 tasks. The total price adds the study once, and the break-even point is the study price divided by the saving per task.

Prices are StreamLake’s list prices on OpenRouter for the model we ran (DeepSeek-V4-Flash-0731): 0.132 USD per million new input tokens, 0.0042 USD per million cached input tokens and 0.396 USD per million output tokens (Appendix C.5).

Every plain and wiki study up to 128k, the 262k wiki study and every parallel or serial study with up to four agents has a lower total cost than no study at the benchmark’s own count of 120 tasks at a 60% score (14 settings; the 95% bootstrap CI over tasks excludes 0 in each; Appendix C.5), against a no-study cost of $4.23. The savings grow with every further task that shares the context.

Retrieval and studying work in different regimes (W2, Q1)

Our focus is how proactive compute scales, which is orthogonal to how retrieval-augmented generation (RAG) performs. Retrieval runs after the task arrives and requires information about the task, while a study runs before it and is inherently task-blind. On AIME 2025 and ARC-AGI the solver already sees the whole source, so retrieval has nothing to add, and a 16k study still raises accuracy from 0.00 to 0.56 (AIME, T=64) and from 0.26 to 0.55 (ARC-AGI, T=4k; Appendix C.6, Figure 3).

Despite these deep differences between the two paradigms, we provide the comparison of a task-blind study with RAG in the following table (Table 5, Appendix C.6; Figure 3). BM25 and dense retrieval see the task and fill the prompt with the most relevant chunks, up to the number of prompt tokens the 16k study gives the solver (about 85k tokens). The file index is task-blind and generates no study tokens.

MethodSees the taskSWE-QA-Pro, T=64T=256T=1kT=4kAIME 2025, T=64ARC-AGI, T=4k
Study (16k)No0.750.770.840.890.560.55
No studyNo0.410.580.790.860.000.26
BM25Yes0.750.780.860.870.00*0.26*
DenseYes0.730.750.850.890.00*0.26*
File indexNo0.510.680.800.900.00*0.26*

SWE-QA-Pro, 16k study. Study and no study: 30 tasks, 2 seeds. BM25, dense and file index: up to 24 tasks, 1 seed. * On AIME and ARC-AGI the solver already sees the whole context, so retrieval adds nothing, and we show the no-study score.

Paired on the same tasks, the task-blind study is never more than 0.01 below task-aware retrieval, and it beats the file index at T ≤ 1k. It also leaves less work for test time. At T=1k the solver makes 2.7 tool calls per task with the study, against 5.2 with BM25 and 10.4 with no study. We include this comparison for completeness, since the question of the paper is how the gain scales with proactive compute.

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Agents can adaptively choose their budgets (W2, Q2)

The study prompt is the same for every offline benchmark and model, and only the context changes (Appendix A; prompts in Appendix H.1), with no per-task tuning (the tool benchmarks add a one-line tool description). On the repository benchmarks the agent already chooses every file it reads. We control the budgets with budget forcing from s1 (Muennighoff et al., 2025), because it sets the raw compute on both axes precisely, which lets us measure how each budget scales. Forcing also gives finer control than effort levels. On a SWE-QA-Pro question, medium effort spent fewer reasoning tokens than low (5,387 against 8,542), so effort levels cannot set a budget.

We keep the study budget fixed so that we can measure how the gain scales with it. The agent already decides when to stop in two places: the uncapped solver stops when it answers, and online, the uncapped agent decides how long to work during each wait. To test whether the solver can decide for itself, we removed forcing in two ways. In both, the study still cuts test-time work at about the same accuracy on SWE-QA-Pro and StudyBench (the first test is in §7.1). First, we let the solver choose its own length. On SWE-QA-Pro, a 16k study still cuts its tool calls from 46.9 to 4.5 per task at the same accuracy (Figure 8 right), and across SWE-QA-Pro, StudyBench and Harvey LAB the unforced point lies within 0.08 of the forced curve at all 24 budgets (P = 0 to 64k) we measured (mean absolute gap 0.04; a new analysis for this response). Second, we set compute only with the model’s reasoning effort. At effort off and low, as with the uncapped runs, a study still cuts tool calls at about the same accuracy (SWE-QA-Pro, effort off: 69.3 to 7.4 per task; new runs for this response). So the study still cuts test-time work without per-task prompt tuning and without forcing.

Efficiency gains hold on all benchmarks (W3, Q4)

We use the math splits as a controlled example, following Sleep-time Compute (Lin et al., 2025), in which the context nearly determines the task. Furthermore, our headline results hold on the other benchmarks too, covering 14 unique repositories on SWE-QA-Pro, 2 code repositories on StudyBench, and a 134M-token legal filesystem from Harvey LAB. On these benchmarks, the task is less predictable, so the study explores broadly and records file locations and observations instead of an answer (§4, Example 2), and the gains still hold. For example, with a 128k study, the solver at T=64 matches a no-study solver with a 15x larger budget on SWE-QA-Pro and 7x on StudyBench (§4.1; Figure 1 for SWE-QA-Pro). On Harvey LAB, E rises from 0.319 without a study to 0.589 with a hierarchical summary (§6; Table 11). The revised introduction leads with these results, and Figure 1 now shows SWE-QA-Pro.

Clarifying Figures 2, 4 and 7 (W1, Q5)

Thank you for your suggestion. We have revised these figures since submission. Figure 2 is now Figure 3. Figure 4 (Figure 5 in the revision) now shows, at 32 seeds per arm, the share of sessions that beat the median human over time, with the reactive agent drawn on top so that every budget reads against it. In Figure 7 (now Figure 8) the left panels show 4 of the 8 study budgets, each with its own marker, and the confidence bands moved to Figure 45 (Appendix G). All figures now use one uniform text size. For a summary per budget, Figure 3 (right; with 95% CIs in Figure 40, Appendix G) gives the efficiency gain for every P, and the appendix gives accuracy over the full P × T grid for each model (Appendix C.8, Figures 16, 20, 24 and 28).

We hope the new measurements address your concerns about cost and baselines, and would be grateful if you would consider raising your score.

Thanks for the detailed response! My concerns have been addressed and I have raised my rating accordingly.

Reviewer_N5rcOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces the concept of proactive compute, unifying different offline and online idle-time compute methods. Experimental results demonstrate that utilizing proactive agents in both offline and online settings improves efficiency, performance, and reduces downstream tool call use and reasoning tokens. The paper also demonstrates the effectiveness of multi agent studying for scaling agentic systems to large contexts.

Critical strengths
  • Thorough experimental design in terms of the breath and relevancy of tasks chosen. Performance gains of proactive agents for the most part seems convincing (moreso with the offline and multi agent setting than the online settings). Overall I am convinced that one should probably use proactive rather than reactive agents for most tasks.
Critical weaknesses
  • It is unclear after reading the paper what exactly the contribution of this paper is compared to the previous works that also study offline and online proactive agents [1,2]. What exactly distinguishes this work from these previous works? I would be willing to raise my score if this question is satisfactorily answered.

[1] Sleep-time Compute: Beyond Inference Scaling at Test-time, Kevin Lin, Charlie Snell, Yu Wang, Charles Packer, Sarah Wooders, Ion Stoica, Joseph E. Gonzalez, https://arxiv.org/abs/2504.13171

[2] IdleSpec: Exploiting Idle Time via Speculative Planning for LLM Agents, Daewon Choi, Kyunghyun Park, Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Jinwoo Shin, Aram Galstyan, https://arxiv.org/abs/2605.22154

Questions for authors
  • See question from Critical Weaknesses section.

  • In Figure 4, why do some proactive agent settings do worse than the reactive baseline?

  • Is it beneficial to combine techniques from both the offline and online settings? I.e. is it beneficial to combine both offline studying with online anticipatory planning?

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

Thank you for the review and for the direct question. We are glad you found the experimental design thorough and are convinced that “one should probably use proactive rather than reactive agents for most tasks”.

We contribute a scaling study across settings, models and budgets (W1, Q1)

We view the core contribution of our paper as a study of how proactive compute scales, rather than a single method for using it. We show that it works in varied and realistic settings (§4–§6), with large modern reasoning models (four models, up to 816B parameters), and we analyze how proactive compute helps (§7).

We treat the settings of both works as a unified problem, compute that an agent spends while it waits. Beyond this similarity in setting, there are deep differences in the range and control of compute budgets, the number of realistic settings, and analysis of why it helps. Three of our results appear in neither paper. (I) We sweep the proactive budget at the token level in both idle periods, independently of the test budget. (II) We study contexts larger than one context window. (III) We find evidence that the study replaces test-time tool calls. The table below compares the three papers side by side.

Sleep-time ComputeIdleSpecOurs
Idle periodBefore the queryDuring tool callsBoth
Models (parameters)GPT-4o, GPT-4o-mini, o1, o3-mini, Claude 3.7 Sonnet, DeepSeek-R1 (671B, 37B active)Gemini-2.5-Flash, Gemma4-E4B (4.5B effective, 8B total), Qwen3.5-4B (4B)DeepSeek-V4-Flash-0731 (284B, 13B active), Qwen3.8-27B (27B), Muse Spark Open (816B, 42B active), gpt-5.6-luna
How proactive compute is variedA few levels: up to 10 parallel samples, or o1/o3-mini reasoning effortDrafts until the tool returns and keeps up to 5 plans (plan budget not swept)9 token-level study budgets from 0 to 128k, and 1k to 16k tokens per wait
Test-time budgetSwept in a few coarse levels (verbosity prompts for GPT-4o/4o-mini, API reasoning settings for o1, o3-mini and Claude 3.7, s1 budget forcing for DeepSeek-R1)Not swept7 token-level budgets from 64 to 64k, swept independently of the proactive budget
BenchmarksStateful GSM-Symbolic, Multi-Query GSM-Symbolic and Stateful AIME (math, built from existing sets), plus 1 custom benchmark: “SWE-Features” (33 tasks)GAIA, FRAMES and MLE-Bench Lite (MLE-Bench Lite: 22 tasks, 1 seed)AIME, HMMT, ARC-AGI-1 and MIR-Bench, plus StudyBench (2 repositories), SWE-QA-Pro (14 repositories), Harvey LAB (a 134M-token legal filesystem), 8 MLE-bench tasks with 32 seeds
Contexts larger than one windowNot testedNot testedParallel, serial and hierarchical studies
Why it helpsA section analyzes how predictable a query is on Stateful GSM-SymbolicAblations on their method of drafting, aggregation and plan selection, and idle-time utilization and accuracy by idle-time lengthTool-call replacement, uncapped solver, compression, planted-error tests

Sleep-time Compute introduced the offline setting and showed that study compute improves performance, amortizes across queries and helps most when the query is predictable from the context. We build on it with the realistic benchmarks, modern reasoning models and token-level sweeps in the table. We also scale the study past one context window (§6.2; positioning in Appendix B). On Harvey LAB, eight parallel agents raise E (our metric for test-time token efficiency) from 0.319 without a study to 0.585, and a hierarchical summary reaches 0.589, above RAG (0.458) and an RLM briefing (0.515) (Figure 7; Table 11 in Appendix E).

We also analyze how studying helps in realistic settings (the study replaces test-time tool calls, so the solver reads and thinks less; §7.1, Figure 8), which Sleep-time Compute does not test.

IdleSpec proposes a specific method of spending proactive compute while tool calls run.

  • In the online setting, we study a general setup instead of proposing one method. The agent may do any work during each wait, not only draft plans.
  • IdleSpec drafts plans until the tool returns, keeps up to five per wait without sweeping this budget, and reports one seed on MLE-Bench Lite with Gemini-2.5-Flash (its 4–8B models run only on GAIA and FRAMES). We sweep the tokens per wait from 1k to 16k on eight MLE-bench tasks with 32 two-hour sessions per arm, on three models of up to 816B parameters. The more idle time the agent uses, the sooner it reaches the same performance: at 16k tokens per wait, half of DeepSeek-V4-Flash’s sessions beat the median human after 24 minutes, against 49 for the reactive agent (Table 9, Appendix D.2).

Our paper ties both settings together as proactive compute and shows that it works in realistic settings with large, modern reasoning models.

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Results on Figure 4 and expanding online benchmarks. (Q2)

In Figure 4 of the submitted paper, we used 3 tasks, 30-minute sessions and budgets from 128 to 8k tokens per wait. In that figure, some proactive curves end worse than reactive at 30 minutes. Most of these are on LMSYS, where no budget differed significantly from reactive (28 paired seeds). On Denoise and Statoil Iceberg, the largest budget (8k) led reactive at 10 minutes (p = .022 and p = .008, both significant after Holm correction).

Since submission we have scaled up the online study: 8 core MLE-bench tasks with every arm and 8 more with the reactive and uncapped arms (up from 3 tasks), 2-hour sessions (from 30 minutes), 32 seeds per task for each arm, every session on its own B200 GPU (10,752 B200-hours in total, including the uncapped arm), and three model families (§5 and Figure 5 for DeepSeek-V4-Flash; all three models in Tables 8 and 9, Appendix D.2, and all 16 tasks per model in Appendix D.3). In this setting, averaged over tasks, proactive agents beat the median human sooner at every budget on every model, and at 2 hours every forced budget ends with a larger share of sessions above the median human than the reactive agent (point estimates from Table 9, Appendix D.2, with DeepSeek-V4-Flash 83–90% against 79%, Qwen3.8-27B 63–71% against 52%, Muse Spark Open 90–92% against 86%). On every model and every budget, proactive agents reach 50% of sessions above the median human sooner, and the 95% CI of the time saved excludes 0. The point estimates of the time saved are 13–25 minutes for DeepSeek-V4-Flash, 9–13 for Muse Spark Open, and at least 38–55 for Qwen3.8-27B, where the reactive agent reaches 50% only at 109 minutes. Table 9 (Appendix D.2) gives the 95% CIs for every model and budget. Minutes until half of the sessions beat the median human, averaged over tasks:

ModelReactive1k per wait4k per wait16k per wait
DeepSeek-V4-Flash49363124
Qwen3.8-27B109715654
Muse Spark Open35262322

So with more tasks, longer sessions and more seeds, the proactive agent beats the median human sooner than the reactive agent at every budget on every model.

Combining offline and online studying can create an early lead (Q3)

Based on this suggestion, we ran a small pilot with DeepSeek-V4-Flash on three MLE-bench tasks (4 pairs each), with a 64k study of the dataset before an online proactive session, against the online agent alone. On us-patent-phrase-to-phrase-matching, the agent that studied the dataset before had an early lead of 0.66 against 0.11 (mean Pearson correlation, the task's metric) at 10 minutes in all 4 pairs. Pooled over the three tasks, the agent with the study also leads at 10 minutes (p=.013). So in this small pilot, the offline study gives the agent an early boost in the session. Online proactive compute alone already reaches the median-human score sooner than a reactive agent on every model (§5; Tables 8 and 9, Appendix D.2).

We hope these answers resolve your concern about our contribution, and we would be grateful if you would consider raising your score.

Thank you for your thorough response. My concerns have been addressed by the authors' rebuttal and I will accordingly raise my rating from 2 --> 3.

Reviewer_fkE2OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

AI was used to rephrase/polish wording. The reviewer wrote the original assessment from scratch and reviewed the final wording.

Original reviewer-written assessment supplied as input:

Flag For Ethics Review

No.

Critical Weaknesses

  1. The claimed cost frontier does not yet reflect the full cost of delivering and maintaining preparation. The main accounting counts only generated tokens, even though the solver receives the study and may reread a growing history across calls. Appendix B.3's discussion of prefix caching and relative token prices helps explain this choice, but it is no substitute for measured total prompt/cache usage, latency, and cost. The savings projected for 1,024 questions are extrapolated from at most 30 questions per context and assume stable reuse. In the online setting, the proactive agent generates additional tokens within the same time budget, so the result shows better use of latency rather than efficiency under equal compute. The paper should keep these distinctions central and provide at least one measured end-to-end cost analysis on a realistic shared repository or document workload.

  2. For the realistic offline tasks, no control separates useful preparation from simply making source information available earlier. As the paper itself notes, the math contexts contain almost the entire problem, and there is no retrieval baseline that loads files without a study. Repository studies also carry tool outputs, so much of the benefit may come from moving file reads into the prompt rather than from anticipatory reasoning. A passive retrieval/cache baseline that delivers comparable information would clarify the additional value of studying. This matters all the more because realistic environments are few: StudyBench uses one repository, SWE-QA-Pro five repositories, and Harvey a single document store. Adding questions and repeated seeds within these environments does not provide the breadth that independent environments would, and the multi-agent efficiency curves rest on one seed.

Questions For Authors

  1. For a repository or Harvey workload, could you report total input, cached-input, and output tokens together with measured cost and query-time latency, and recompute the study's break-even point under those measurements?
  2. How does study compare with a baseline that preloads or retrieves comparable source material without generating a study trajectory? Does the reduction in reasoning and tool calls under the natural budget hold up in this comparison?
  3. To what extent do the wiki and parallel-study gains depend on independent study seeds and on which environments are selected? Please distinguish question-level uncertainty conditional on one study from variation across independently generated studies.

Rating

3: Weak acceptance.

I lean toward acceptance, since the paper offers a substantial empirical investigation of when preparation replaces downstream reasoning, and its interventions and natural-budget experiment help explain the effect. Its contribution lies mainly in carefully characterizing and extending earlier idle-time-compute ideas, not in introducing proactive computation itself. My main reservations relate to complete resource accounting and to the lack of a retrieval baseline matched on source information. In my view, the current results provide strong evidence of improved query-time behavior when preparation opportunities are favorable, and more limited evidence of general savings in total cost.

Soundness

2: I have minor or fixable concerns about the paper’s soundness.

Paper Summary

The paper investigates computation carried out before a query arrives and while an agent waits for tool results. In the offline setting, study and downstream reasoning budgets are varied independently across nine benchmarks, and the frontier of downstream accuracy against generated tokens is measured. Amortization is also analyzed for the case where several questions share one study. In the online setting, agents reason during tool execution on three MLE-bench tasks under equal wall-clock budgets of 30 minutes. On a 134M-token legal collection, preparation is extended through wiki-style exploration and partitioned multi-agent study. Trace analyses, corrupted-study interventions, compression, and unconstrained downstream reasoning tie the gains to solving problems in advance and gathering information. The strongest supported conclusion is that, when relevant context is already available, preparation can reduce query-time reasoning and tool use, while overall resource savings depend on reuse and on cost assumptions.

Presentation

3: Clear and easy to follow.

Critical Strengths

  1. Thanks to the controlled budget sweeps and follow-up analyses, the study shows more than the fact that extra computation helps. By separating preparation from downstream reasoning, it reveals where preparation substitutes for reactive work and where its gains saturate. The natural-budget experiment stands out: on SWE-QA-Pro, a 16K-token study cuts downstream thinking by more than half and reduces tool calls roughly tenfold at comparable reported accuracy, without forcing the solver into a small reasoning budget. The compression and corrupted-study interventions further show that the retained information and trust in the study are central to the effect.

  2. The paper covers both large environments and actual waiting periods, and it discloses important limitations. The Harvey experiments examine organization and document coverage beyond what a single context window can hold. In the online study, runs are paired over 28 seeds per task under equal wall-clock budgets, with Holm correction applied across the three tasks. At ten minutes, the 8K proactive agent improves Denoise and Iceberg, whereas the LMSYS comparison is not significant. The authors also explicitly acknowledge the near-complete math contexts, the generated-token-only accounting, the missing retrieval baseline, and the limited multi-agent seeds. As a result, the contribution can be assessed as an empirical characterization rather than as an unconditional efficiency claim.

Informativeness

3: I gained a substantially deeper understanding of the problem or question.

Critical strengths

The paper investigates computation carried out before a query arrives and while an agent waits for tool results. In the offline setting, study and downstream reasoning budgets are varied independently across nine benchmarks, and the frontier of downstream accuracy against generated tokens is measured. Amortization is also analyzed for the case where several questions share one study. In the online setting, agents reason during tool execution on three MLE-bench tasks under equal wall-clock budgets of 30 minutes. On a 134M-token legal collection, preparation is extended through wiki-style exploration and partitioned multi-agent study. Trace analyses, corrupted-study interventions, compression, and unconstrained downstream reasoning tie the gains to solving problems in advance and gathering information. The strongest supported conclusion is that, when relevant context is already available, preparation can reduce query-time reasoning and tool use, while overall resource savings depend on reuse and on cost assumptions.

Critical weaknesses
  1. Thanks to the controlled budget sweeps and follow-up analyses, the study shows more than the fact that extra computation helps. By separating preparation from downstream reasoning, it reveals where preparation substitutes for reactive work and where its gains saturate. The natural-budget experiment stands out: on SWE-QA-Pro, a 16K-token study cuts downstream thinking by more than half and reduces tool calls roughly tenfold at comparable reported accuracy, without forcing the solver into a small reasoning budget. The compression and corrupted-study interventions further show that the retained information and trust in the study are central to the effect.

  2. The paper covers both large environments and actual waiting periods, and it discloses important limitations. The Harvey experiments examine organization and document coverage beyond what a single context window can hold. In the online study, runs are paired over 28 seeds per task under equal wall-clock budgets, with Holm correction applied across the three tasks. At ten minutes, the 8K proactive agent improves Denoise and Iceberg, whereas the LMSYS comparison is not significant. The authors also explicitly acknowledge the near-complete math contexts, the generated-token-only accounting, the missing retrieval baseline, and the limited multi-agent seeds. As a result, the contribution can be assessed as an empirical characterization rather than as an unconditional efficiency claim.

Questions for authors
  1. The claimed cost frontier does not yet reflect the full cost of delivering and maintaining preparation. The main accounting counts only generated tokens, even though the solver receives the study and may reread a growing history across calls. Appendix B.3's discussion of prefix caching and relative token prices helps explain this choice, but it is no substitute for measured total prompt/cache usage, latency, and cost. The savings projected for 1,024 questions are extrapolated from at most 30 questions per context and assume stable reuse. In the online setting, the proactive agent generates additional tokens within the same time budget, so the result shows better use of latency rather than efficiency under equal compute. The paper should keep these distinctions central and provide at least one measured end-to-end cost analysis on a realistic shared repository or document workload.

  2. For the realistic offline tasks, no control separates useful preparation from simply making source information available earlier. As the paper itself notes, the math contexts contain almost the entire problem, and there is no retrieval baseline that loads files without a study. Repository studies also carry tool outputs, so much of the benefit may come from moving file reads into the prompt rather than from anticipatory reasoning. A passive retrieval/cache baseline that delivers comparable information would clarify the additional value of studying. This matters all the more because realistic environments are few: StudyBench uses one repository, SWE-QA-Pro five repositories, and Harvey a single document store. Adding questions and repeated seeds within these environments does not provide the breadth that independent environments would, and the multi-agent efficiency curves rest on one seed.

2 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

Thank you for your careful review, and for recognizing that the paper is “a substantial empirical investigation of when preparation replaces downstream reasoning”, that it shows “more than the fact that extra computation helps”, and that “the retained information and trust in the study are central to the effect”. Following your questions, we measured total cost with input and cached tokens on Harvey LAB and ran retrieval baselines testing BM25, dense retrieval, and static file indices. Both analyses are now in Appendix C of the revised paper (Table 4 in C.5, Table 5 in C.6).

Studies lower the end-to-end cost on Harvey LAB (W1, Q1)

We now price every token on Harvey LAB, for the 120 tasks that share its filesystem, at the test budget where the score first reaches 60%, interpolated between grid budgets. The same rule picks the budget for every row. Prices are StreamLake’s list prices on OpenRouter for the model we ran (DeepSeek-V4-Flash-0731): 0.132 USD per million new input tokens, 0.0042 USD per million cached input tokens and 0.396 USD per million output tokens. Cached input counts are the cached tokens that the provider reports for each call. The paper gives these list-price totals in Table 4 (Appendix C.5). The billed study prices, about a third of list prices, are in Table 11 (Appendix E). We priced Harvey LAB in detail because it is the closest to real use, with 120 tasks that share one context.

MethodStudy ($)InputCached inputOutputTotal, 120 tasks
No study024.6M78.9M1.65M$4.23
Plain study, 16k0.02711.3M102.0M1.29M$2.45
Wiki study, 32k0.0535.5M114.0M1.01M$1.66
One parallel study, 128k0.2308.2M31.7M0.86M$1.79

These are the studies of the Harvey LAB case study (§6). Each serves all 120 tasks, and its cost includes its own input (which includes all the tokens generated and seen by the proactive agent). With input and cached tokens counted, these studies break even after 1.8, 2.4 and 10.3 tasks at the 60% target, fewer than the 120 tasks that share the context.

So with every token counted, each study lowers the end-to-end computational cost, even after only a few of the 120 tasks.

Retrieval baselines are different settings; task-blind study matches task-aware retrieval (W2, Q2)

Our focus is how the gains of proactive compute scale with the study and test budgets, and we don’t try to optimize a single method to beat baselines like retrieval-augmented generation (RAG). Additionally, studying and retrieval exist in different settings. Studying happens before the task, while retrieval happens after the task is given. Studying is task-blind, while retrieval needs the task to retrieve relevant context. Even despite these differences, we compare retrieval and studying in two main ways:

First, on AIME and ARC-AGI the solver without a study already has the whole source in its prompt. A 16k study still raises AIME 2025 accuracy at T=64 from 0.00 to 0.56, and ARC-AGI accuracy at T=4k from 0.26 to 0.55 (Appendix C.6; Figure 3). This gain does not come from access to the source material. Instead, it comes from reasoning intelligently over the given material.

Second, despite retrieval and studying being in different regimes and settings, we compare them head to head in the following table. BM25 and dense retrieval see the task and fill the prompt with the most relevant chunks, up to the number of prompt tokens that the 16k study gives the solver (about 85k tokens, its own output plus the tool outputs it read). The file index is task-blind and generates no study tokens.

Method (SWE-QA-Pro)Sees the taskT=64T=256T=1kT=4kTool calls per task, T=1k
Study (16k)No0.750.770.840.892.7
No studyNo0.410.580.790.8610.4
BM25Yes0.750.780.860.875.2
DenseYes0.730.750.850.897.2
File indexNo0.510.680.800.909.3

Study and no study: 30 tasks, 2 seeds. Retrieval and file index: up to 24 tasks, 1 seed.

Without seeing the task, the study is never more than 0.01 below task-aware retrieval when paired on the same questions. At T=1k it needs 2.7 tool calls per task, against 5.2 with BM25, 7.2 with dense retrieval and 10.4 with no study, so the reduction in tool calls holds against retrieval. The table is also in the paper as Table 5 (Appendix C.6). So without seeing the task, a study performs on par with task-aware retrieval and leaves the solver less work at test time, which supports our focus on how the gains of proactive compute scale.

AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Proactive compute turns idle time into earlier results (W1)

We agree with this assessment. Online, proactive compute saves wall-clock time. The agent generates tokens while tools run, in time that would otherwise be idle, and so it reaches a better result sooner (Section 5, and Figure 34 in Appendix D.4 plots performance against generated tokens). Until half of the runs beat the median human, the 16k agent generates 0.96× the reactive agent’s tokens with Qwen3.8-27B, 1.34× with Muse Spark Open and 1.74× with DeepSeek-V4-Flash. With these tokens, half of the 16k agent's sessions beat the median human after 54, 22 and 24 minutes, compared with 109, 35 and 49 minutes for the reactive agent (Qwen3.8-27B, Muse Spark Open and DeepSeek-V4-Flash; Table 9, Appendix D.2).

Environments and study seeds. The gains hold across studies and environments (Q3)

We generated three independent study seeds for each Harvey LAB arm (plain, wiki, parallel and serial; Table 12, Appendix E). For each group below, we compare (1) the variation of expertise between seeds, and (2) whether the comparison between arms changes from set to set. The standard deviations below are the variation across independently generated studies. For comparison, the question-level uncertainty for one study (95% half-width of E) is 0.038–0.042 for plain and wiki studies and 0.044–0.050 for parallel and serial studies (95% CIs in Table 11, Appendix E). On SWE-QA-Pro, the 95% half-width of the gain over the 14 repositories is 0.007–0.011.

  • Harvey plain and wiki studies (120 questions). (1) For the plain studies, E rises over no study (0.319) by +0.057 to +0.094, and the standard deviation across seeds is 0.017–0.038. For the wiki studies, E rises by +0.132 to +0.164, and the standard deviation is 0.006–0.036. (2) Every seed beats no study (smallest gain +0.027), and the wiki study beats the plain study at every study budget in every set (15 of 15 comparisons).
  • Harvey parallel and serial studies (75 questions, each now with 3 seeds). These are the 75 questions that all three sets answered, and Table 12 in Appendix E gives the three sets. (1) E rises over no study by +0.113 to +0.233, and the standard deviation across sets is 0.029–0.060 (0.029 for eight parallel agents, an eighth of their gain). (2) Every arm beats no study in every set, and eight parallel agents beat one study in every set (pooled +0.115 [+0.079, +0.151], p<.001).
  • SWE-QA-Pro (140 questions over 14 repositories, each with its own study, and 3 study seeds). (1) For study budgets from 4k to 128k, E rises over no study by +0.058 to +0.083, and the standard deviation across seeds is 0.003–0.008, about a tenth of the gain. (2) Every seed beats no study at every study budget (Table 13, Appendix E).

So the gain does not depend on the particular study that was generated, since every independent study beats no study, and the main comparisons (wiki over plain, eight parallel agents over one study) hold in every study seed.

Since our submission, we have added a fourth model, 14 repositories (from 5) and up to 400 questions per benchmark (Table 3, Appendix C.2; Appendices C.7–C.8). We hope these measurements address your reservations about resource accounting and about variation between studies, and we would be grateful if you would take them into account in your final assessment.

Reviewer_DYeXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper systematically explores scaling proactive compute for LLMs, where the agentic system waits for responses or local tools to run. This study examined LLM performance with preparation and downstream reasoning budgets across nine offline benchmarks, and LLM reasoning performance while tools run for engineering tasks. Overall, these experiments revealed interesting findings, including that reusing studies across questions can reduce total generated token costs, and that reasoning. At the same time, tools run achieves better scores under equal wall-clock budgets. In addition, this paper explores wiki-style and multi-agent study strategies for a law benchmark (Harvey LAB), which proved more efficient than plain study at every tested budget.

Critical strengths
  1. This paper explores the test-time compute with proactive compute, which is intuitive and makes sense.
  2. This paper aims to evaluate proactive compute across math reasoning, engineering, and a law domain application, which is comprehensive.
  3. The experiment is also conducted on both strong public available and closed source models across many seeds, which demonstrates the broader applications of this concept.
  4. The analysis of idle time (Figure 5) is simple and effective in demonstrating the potential of proactive compute in real applications.
Critical weaknesses
  1. The paper is generally hard to follow. Each evaluation seems disconnected from the others and differs in terms of setup and metrics. However, they fall under the same proactive compute theme. It would be ideal to better organize the experiments.
  2. In Section 4, for the offline setting, it is unclear what proportion of the trajectories falls under the desired budget. Will appending “Keep reasoning. Do not answer yet” influence the evaluation results compared to the prompt in the one-shot case?
  3. It is unclear what the proactive compute actually does in practice. It would be helpful to provide one or two cases to analyze why there is an efficiency gain for each of the experiment conducted.
  4. Regarding context reuse in Figure 3, it is unclear how much of the context is reused. The analysis is incomplete without the cost of context reuse tokens.
Questions for authors

See weakness

3 comments
AuthorsOpenReview ↗

Rebuttal by Authors 1/2

Thank you for the review. We are glad you found the evaluation across math, engineering and law “comprehensive”, and the idle-time analysis “effective in demonstrating the potential of proactive compute in real applications”. Below, we answer each point in turn: how the sections fit together (W1), budget forcing and the continuation prompt (W2), the cost of context reuse (W4), and concrete examples of what a study does (W3).

Organization of paper sections (W1)

While the sections use different setups and metrics, all sections of the paper address one question, how much test-time resource an agent saves after it spends proactive compute. Proactive compute itself can be spent in two ways, before the task (offline) and while tool calls run (online), so we give the offline (§4) and online (§5) settings one section each. Additionally, §6 applies the offline setup to a large context. The table below outlines the sections:

SectionIdle periodSetupWhat we varyMain metric
Offline (§4), 9 benchmarksBefore the taskStudy a contextStudy budget P, test-time budget TScore against test budget T, test-time efficiency E
Online (§5), 8 MLE-bench tasksWhile a tool runsKeep working while a tool runsProactive tokens P per waitShare of sessions that beat the median human, against time
Harvey LAB (§6)Before the taskStudy a large contextStudy budget P, test-time budget T, study method, number of agentsScore against test budget T, test-time efficiency E

Every section measures how much scarce test-time resource an agent needs to reach a score once it has spent proactive compute:

  • Offline, that resource is test-time tokens: we report score against the test budget T and the test-time efficiency E (defined in §3, weights in Appendix C.4), a one-number summary of that curve adapted from the expertise metric of Machine Studying (Li et al., 2026).
  • Online, it is wall-clock time: we report the share of sessions that beat the median human against time, and the minutes until half of them do.

§6 uses the offline setup, but scales the study past one context window and varies the study method and the number of agents.

Budget forcing is effective and valid (W2)

Through budget forcing, nearly every run reaches its budget:

ModelStudies that reach 95% of PSolver calls that reach 95% of T
DeepSeek-V4-Flash99.98% of 17,23299.87% of 190,701
Qwen3.8-27B99.75% of 17,22599.92% of 190,701
Muse Spark Open98.58% of 17,23296.37% of 190,701
gpt-5.6-luna97.68% of 4,05697.57% of 34,019

On every model, at least 96% of studies and solver calls reach their budget, and the median call reaches 100%.

We adopt budget forcing with appended continuation prompts from s1 (Muennighoff et al., 2025). It allows us to set the raw compute on both axes precisely, so we can sweep the study and test budgets independently. The general studying prompt is the same for every benchmark and model (Appendix H.1).

We also ran three tests without budget forcing.

  • Letting the solver pick when to stop. In the ‘uncapped solver’ experiment, we allow the solver to choose its test-time budget, so we do not apply budget forcing after the model stops. On the three agentic benchmarks, none of the paper's 24 unforced points of §7.1 (Figure 8 right; P = 0 to 64k, 8 per benchmark) lies more than 0.08 below the forced curve at the same number of thinking tokens, and on SWE-QA-Pro all 8 lie above it by 0.03–0.06. So a solver that picks its own budget lands close to the forced curve, and gains from proactive compute don’t rely on budget forcing.
  • Varying reasoning using effort levels. We also removed forcing from the solver completely, and set its compute only with the API’s reasoning effort. At most points, the results stay close to the budget-forced ones: on the three agentic benchmarks, 32 of the 35 points lie within 0.08 of the forced curve at the same number of thinking tokens, and on AIME all 20 points lie above it. With reasoning off, a 16k study still cuts SWE-QA-Pro tool calls from 69 to 7 per task. So the gains also hold when only the effort setting sets the solver's compute.
  • One fixed continuation prompt. The sentence you quote, "Keep reasoning. Do not answer yet.", is the first of the solver's three rotating continuation prompts (§3; Appendix H.2), and the solver's prompts are the same in both versions of this test. We replaced the study's three rotating continuation sentences with that one sentence on AIME, SWE-QA-Pro and StudyBench. The solver's score changed by −0.07 to +0.04 (1 seed, 18–30 questions). Every 95% CI includes 0, so the continuation prompt does not measurably change the results.
AuthorsOpenReview ↗

Rebuttal by Authors 2/2

Considering context reuse, studying still saves on end-to-end costs (W4)

Following your suggestion, we now price every input, cached-input and output token. The study is generated once and is a shared, cached prefix for every task that shares its context (120 tasks on Harvey LAB). With input, cached-input and output tokens priced on Harvey LAB, at the test budget where the score first reaches 60%, interpolated between grid budgets:

Harvey LAB, 120 tasksStudy priceAnswer price per questionAvg output tokens per questionInput tokensCached input tokensOutput tokensTotal priceSaving
No study$0$0.035313.7k24.6M78.9M1.65M$4.23–
Wiki study, 32k$0.053$0.01348.4k5.5M114.0M1.01M$1.66$2.57 (61%)

The full table is in our reply to fkE2 and in Table 4 (Appendix C.5). We price input, cached-input and output tokens at 0.132, 0.0042 and 0.396 USD per million tokens (StreamLake’s list prices on OpenRouter for DeepSeek-V4-Flash-0731). While the cached-input price depends on the provider, a 32k wiki study on Harvey LAB would lower the total cost for cached-input prices up to about 17 times more than this, about 54% of the new-input price, whereas most providers of this model price cached inputs at about 21% of the input price (the median over 27 providers on OpenRouter). One parallel 128k study lowers the total cost at any cache price, because it also lowers the number of cached tokens that the solver reads.

So counting the reused context does not change the result. With input tokens priced, the plain and wiki studies break even after 1.8 and 2.4 tasks. This is earlier than the 9 to 11 tasks found when only output tokens are counted (16k study, §4.3, Figure 4), because the solver without a study explores more and reads more new input tokens. The wiki study cuts uncached input from 24.6M to 5.5M tokens.

Proactive compute in practice (W3)

Below are three examples (Appendix C.3), from a repository, an algorithmic puzzle and a legal task.

  • Repository (SWE-QA-Pro, CEKit, P=4k, T=64).
    • Context: the CEKit repository.
    • Task: how does CEKit’s descriptor override system coordinate between configuration loading, descriptor merging, and path resolution when processing override files?
    • Study: within its 4k tokens, it read lines 45–50 of cekit/generator/base.py, which pass “os.path.dirname(os.path.abspath(override))” to each Overrides object.
    • Result: With the study, the solver cites these lines: it “wraps the result in Overrides(tools.load_descriptor(override), os.path.dirname(os.path.abspath(override)) )”. Without it, the solver says it “calls descriptor.apply_paths(root_dir)”, and no such function exists in the repository. Mean score over 3 seeds 0.43 → 0.82, with no tool calls in either arm.
  • Procedure discovery (ARC-AGI-1, task 4cd1b7b2, P=1k, T=64).
    • Context: three 4×4 example pairs of input and output grids.
    • Task: a new 4×4 test input.
    • Study: it found that the “Output grids are Latin squares of order 4” and that “Every non-zero input value is preserved in the same position in the output.”
    • Result: With the study, the solver is correct on 3 of 3 seeds. Without it, 0 of 3: every no-study answer overwrites given cells (5, 3 and 6 of 10). So the study finds the rule, and the solver only has to apply it.
  • Harvey LAB (task 026, 16k wiki study, T=1k).
    • Context: the firm’s 134M-token document store, 266 matters and 9,288 files.
    • Task: “What’s our latest Banking & Finance matter that included a revolver, so I can pull the mechanics?”
    • Study: it built one record per matter from the engagement letters, for example “1019-00002 | 35f | engdate=October 20, 2025”.
    • Result: With the study, all 3 seeds name 1019-00002 with 0 tool calls. Without it, the solver makes 4 or 5 tool calls, and all 3 seeds pick 1038-00002, an older revolver matter (closed May 19, 2023). Mean score over 3 seeds 0.00 → 0.56. So the study's records of matters replace the solver's search of the store.

In each case the study does part of the task’s work before the task arrives. On the tool benchmarks this shows up as fewer test-time tool calls: with a 128k study at T=64k, the solver makes 10x fewer on StudyBench, 4x fewer on SWE-QA-Pro and 1.7x fewer on Harvey LAB, while accuracy changes by at most 0.03 (§7.1, Figure 8 left). Online, the saving is in time. On the MLE-bench task Leaf Classification, with 16k proactive tokens per wait, the median session writes its first valid submission file at minute 13, against minute 55 for the reactive agent (Table 7, Appendix D.2).

We hope the table outlining the section organization, as well as the updated cost accounting, make the paper easier to follow and address your concerns, and we would be grateful if you would consider raising your score accordingly.

Thanks for the newly provided experiment results on the budget forcing, which address my major concern. The detailed analysis of context reuse tokens further demonstrates the potential application of the proposed method. Thus, I am raising my rating to 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General comment by Authors

We thank the reviewers for their feedback. Since submission, we have expanded the paper with a fourth model, up to 400 tasks per benchmark, 3 seeds and a larger online study.

In our replies, we include these main analyses:

  1. Cost with input tokens (fkE2, q9ix, DYeX). On Harvey LAB we price the input, cached-input and output tokens of every call. Every plain and wiki study up to 128k tokens and every parallel or serial study with up to four agents lowers the end-to-end cost of answering the 120 tasks at a 60% score. A 32k wiki study cuts it by 61%, from 4.23 USD to 1.66 USD (Table 4, Appendix C.5).
  2. Retrieval baselines (fkE2, q9ix). Retrieval requires knowing the task, while studying is task-blind, so the two work in deeply different regimes and settings. For completeness, we compare them head to head. On SWE-QA-Pro, a 16k study that never sees the task is never more than 0.01 below task-aware BM25 and dense retrieval. On AIME 2025 and ARC-AGI-1, where the solver already sees the whole source, the study raises accuracy from 0.00 to 0.56 and from 0.26 to 0.55, while retrieval cannot add anything (Table 5, Appendix C.6).

Since submission, we have updated the paper with:

  1. More models, tasks and seeds. Four models (DeepSeek-V4-Flash, Qwen3.8-27B, Muse Spark Open and gpt-5.6-luna), up to 400 tasks per benchmark and 3 seeds (Table 3, Appendix C.2). SWE-QA-Pro now covers 14 repositories (from 5), and StudyBench both of its repositories.
  2. A larger online study (N5rc, fkE2). 16 MLE-bench tasks, with 32 two-hour sessions per arm (8 tasks with all 5 arms, displayed in main paper) on three models. With 16k tokens per wait, half of DeepSeek-V4-Flash’s sessions beat the median human after 24 minutes, against 49 for the reactive agent (§5, Appendix D.2).
  3. Study variance (fkE2). Every Harvey LAB study arm beats no study in each of three independently generated study sets (Table 12, Appendix E).

We thank the reviewers again for their constructive feedback and address each reviewer’s points individually below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.