acceptodds
Under review as a conference paper at ICLR 2027

Regime-Dependent Inference-Time Compute Allocation for Agentic Systems: When Deeper Reasoning Wins

Abstract

Longer reasoning, repeated sampling, revision, and communication have each been advocated as ways to improve language-model performance through their composition into agentic systems. These competing prescriptions leave a practical question: which use of compute is optimal for a given model, task, and compute budget? We organize this choice around the generation, preservation, and selection of correct answers, and compare 96 configurations of one model on four reasoning benchmarks. At a matched budget of 16,384 reasoning tokens, depth gives the highest observed accuracy in three harder groups defined retrospectively by short-run success. In the frequent-success group, voting reaches the observed ceiling at one-quarter of that expenditure. We then ask whether allocation can be prescribed before testing larger budgets on a new problem. A logistic rule trained on graded development outcomes at several depths chooses depth–width allocations using five short probes per new item. In an exploratory comparison on 200 unseen problems, it reaches 70.0% accuracy, versus 70.5% for fixed depth. The results support conditional allocation guidance and show how a calibrated rule can act before a new problem's scaling curve is measured.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, Agentic AI, Reasoning, Test Time Compute, Pass@k

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_j9dPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper explores what to do when giving a language model a fixed inference budget: should it think longer in a single generation, try multiple independent attempts, or have multiple agents repeatedly revise and discuss. The authors propose an adaptive method and run some experiments to validate it, along with some trivial theoretical analysis.

Critical strengths

The author proposes their own method and conducts a set of experiments, supported by several figures illustrating the approach and results. Additional detailed results are provided in the appendix.

Critical weaknesses
  1. The paper's theory and modeling have major issues. The authors use a lot of complex mathematical notation in places where it is completely unnecessary. At the same time, in places that genuinely require mathematical understanding, they fail to do so, demonstrating a real lack of familiarity with the field. In Line 140, the authors discuss reasoning cost, but they only consider tokens and completely fail to factor in FLOPs as an approximation for actual compute.

  2. The proposed method is not novel; plenty of previous work has already addressed this much more comprehensively.

  3. The method requires probing every model a certain number of times for each problem to decide which approach to use. The authors should consider this cost.

  4. The scope and scale of the experiments are far too small to support their claims.

Questions for authors

NA

3 comments
AuthorsOpenReview ↗

Response to Reviewer j9dP

Thank you for your review. We address the compute and probing concerns below and request the specific information needed to assess the remaining objections. Brackets denote 95% confidence intervals.

1. Mathematical presentation and soundness.

You state that our theory and modeling have “major issues.” Apart from the compute-accounting concern addressed below, however, the review does not identify a mathematical statement, assumption, inference, or proof step alleged to be incorrect.

As per ICLR reviewer guidelines can you “Provide supporting arguments for your recommendation.” It would help us addressing if you could identify each mathematical issue and how it affects our conclusions. Likewise, please specify which mathematical details you consider unnecessary and which phenomena you believe require analysis that is absent.

These particulars are necessary for us to evaluate and address the stated soundness concern. We are ready to examine each specific issue and make any correction it warrants as soon as we know!

2. Reasoning tokens versus FLOPs.

The manuscript explicitly states in §8, lines 385–386:

“Budgets match reasoning tokens rather than total FLOPs or deployment cost, since histories, peer messages, and answer-generation calls incur different costs.”

The distinction between reasoning-token expenditure and total computation is therefore explicitly recognized. The OpenRouter runs used for the main experiments did not provide FLOP telemetry, and the paper states the resulting limitation clearly in lines 385-386.

To address the substantive compute question, we have now added 4,992 calls comparing depth, voting, and debate on dense Qwen3-8B and Qwen3-32B under natural stopping, with estimated FLOPs accounting for both prefill and decoding (Experiments 4–5). Depth is more accurate while voting and debate use 1.68–2.93× as many estimated FLOPs, or 1.57–2.68× with shared-prefix caching.

Thus, for these dense models, accounting for prefill and decoding preserves the ordering: the less accurate alternatives also consume more estimated computation. If you consider this comparison insufficient to address your objection, please identify the missing cost term or the specific comparison that remains inadequately controlled.

3. Novelty relative to prior work.

The ICLR reviewer guidelines require to substantiate your statement that previous work addresses our contribution more comprehensively, identifying the relevant papers and specifying which of our contributions they already establish. We apologize if we have overlooked relevant work and will correct any missing attribution. However, in our knowledge our contributions do not appear in the related literature.

4. Per-problem probe cost.

The manuscript explicitly states in §6, lines 322–323, that “probes add 5A to solver cost,” and distinguishes the reported tradeoff from an equal-total-compute comparison.

The supporting experiment demonstrates predictive signal on held-out problems: the additive model, which includes probe and provider features, achieves a Brier score of 0.179 versus 0.202 for the population-only fit (§E.2). Each routing decision uses five 1,024-token solver calls, adding 5,120 reasoning tokens, approximately USD 0.018, and 33 seconds of modeled latency.

We have now charged all five probes against a shared 16,384-token ceiling. On 200 held-out problems, fixed depth achieves 70.5% accuracy versus 64.0% for the allocator, a 6.5-percentage-point advantage [1.5, 11.5] (Experiment 2). This reports the consequence of paying for the probes within the available budget, including the resulting advantage for fixed depth.

Can you please tell us any component of the per-problem probing cost that you believe remains unaccounted for?

5. Experimental scope and scale.

You state that the experiments are “far too small to support their claims.” Please specify which claim you consider unsupported and what experimental scope would provide compelling evidence for that claim?

The expanded evaluation covers seven models from five families, 96 problems from four benchmarks, and two repeats per item, together with additional 4,096–65,536-token budget sweeps on DeepSeek and Claude. We believe this substantially strengthens the evidence for the conclusions within the evaluated settings.

We would appreciate if you could reassess the scope objection against this expanded evaluation. If you maintain it, can you please identify the remaining gap? E.g., model coverage, task coverage, sample size, or a particular comparison. Also, would you be willing to explain which conclusion depends on the additional evidence you would like to see. This would allow us to address concretely!

We hope you update your assessment in light of these clarifications and additional results, and state the specific grounds for any concerns you retain. We remain ready to address those grounds directly :)

Hello! As the deadline is approaching, we'd be greatful if you could find the time to comment on this :) We genuinely believe that we could be actionable about your comments/weaknesses if you could be more precise. Gives us a change to convince you the worth of our paper :)

Good luck with your own submissions,

Best,

The authors

Thank you for the response. The new probe-cost experiment shows that fixed depth (70.5%) outperforms the probe-based allocator (64.0%) once probe costs are charged, which reinforces my concerns. I will keep my score.

Reviewer_C94HOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates how to optimally allocate computational resources during the inference phase of LLMs, comparing three strategies: increasing reasoning depth within a single inference pass, employing multi-agent deliberation, and performing multiple rounds of sampling under a fixed computational budget. Through a well-designed methodology and comprehensive experimental analysis, the authors show that deepening the reasoning process within a single inference pass is the most effective strategy. Furthermore, they propose a practical method for selecting the optimal inference strategy at deployment time and provide empirical evidence of its effectiveness. Overall, the study is thorough and well motivated. However, a significant limitation is that the experiments are conducted using only a single model, raising concerns about the generalizability of the findings.

Critical strengths

The paper addresses a timely and important research question: how to optimally allocate computational resources at inference time, whether by increasing reasoning depth, leveraging multi-agent frameworks, or performing multiple rounds of sampling. Beyond establishing the relative effectiveness of these strategies, the authors propose a practical, low-overhead routing method that enables adaptive strategy selection. This combination of empirical analysis and practical applicability makes the work both relevant and potentially valuable.

Critical weaknesses

The most significant limitation is the reliance on a single open-source model with 27B parameters. This substantially limits the generalizability of the conclusions, as it remains unclear whether the observed trends extend to state-of-the-art closed-source models or other model families. The reported results may be specific to the Qwen family or even to the particular Qwen3.8-27B model evaluated in the study. Additionally, although the router is an interesting and potentially effective design, it is unclear whether the allocator must be retrained when the framework is applied to tasks in different domains. Addressing this question would help clarify the method's adaptability and practical deployment costs.

Questions for authors
  1. The authors should to evaluate a broader range of models to assess whether the findings generalize across different model families, parameter scales, and model types, including state-of-the-art closed-source models where feasible?
  2. Does the proposed allocator need to be retrained when the framework is applied to tasks in different domains or tasks? If so, how much additional training data and computational resources are required?
  3. After accounting for the computational overhead of the five probes used for initial exploration, does the proposed allocator remain more computationally efficient than a fixed deep-reasoning strategy? Besides, the authors need to compare the two approaches in terms of accuracy under an identical total computational budget, including the overhead incurred by the allocator? Such an evaluation would provide a fairer assessment of the allocator's practical benefits.
3 comments
AuthorsOpenReview ↗

Response to Reviewer C94H

Thank you for your careful reading and encouraging assessment. Let us know if below we address all your concerns and questions conclusively! We are very willing to further work with you to clear up any further question! You identified model coverage as your main reservation and asked specific questions about transfer and deployment cost. We have completed studies addressing each question and explain below what they establish.

1. Generalization beyond the original model. You asked us to test other families, parameter scales and proprietary models. We have done so: the evaluation now covers seven models from five families (Experiments 1, 3, 3b and 4).

At nominal 16,384-token allocations, depth exceeds voting and debate by 10.9–14.6 percentage points on Qwen3-8B/32B. On DeepSeek-V4.1 Flash, it leads voting by 15.6 points and debate by 19.3 points; on Claude Haiku 5.5, it leads debate by 15.1 points. On Gemini 3.8 Flash with room to answer, it leads voting by 5.7 points [1.6, 10.4], with mean realized reasoning differing by 13%. Brackets denote 95% intervals.

Claude voting, Gemini's shared-cap comparisons and both GPT-6.1 Terra contrasts remain inconclusive. We will report them alongside the positive results. Provider effort settings and natural stopping do not ensure equal realized tokens or FLOPs.

We also swept DeepSeek and Claude from 4,096 to 65,536 tokens (Experiment 3c). Depth leads voting/debate by 19.8–25.0 points at 4,096 tokens, with gaps narrowing at larger budgets. The advantage therefore appears across several additional families and depends on the available budget.

These experiments directly answer whether the finding is confined to the original Qwen checkpoint. Do they resolve your concern about model coverage? If a particular generalization claim still seems unsupported, please identify it and the comparison you think is missing.

2. Retraining and data requirements. You asked whether applying the allocator to another task requires retraining. In our four benchmark-transfer tests, omitting target-benchmark training labels did not reduce aggregate decision accuracy: 64.0% under a 16,384-token all-in ceiling and 70.0% under a 21,504-token ceiling, matching the full fit (Experiment 6).

Adding up to 24 labeled target problems improved probability calibration in all four folds without improving accuracy. For example, Omni-MATH Brier score fell from 0.286 to 0.261. Generating graded outcomes for 24 target problems cost approximately USD 5, and refitting took approximately 0.9 seconds. These are recalibration costs after source training; the original full fit uses 96 development problems and 1,192 allocation outcomes.

Thus, for the tested transfers, target labels were unnecessary to retain aggregate accuracy but useful for calibration. We do not infer data requirements for arbitrary new domains or models from these four benchmarks on one solver. Does this answer your questions about the tested transfer setting and its cost? Please let us know which part remains unclear.

3. Accuracy and efficiency after charging all five probes. We have performed the comparison you requested. On 200 held-out problems, charging the probes against a common 16,384-token ceiling gives 70.5% accuracy for fixed depth versus 64.0% for the allocator, a 6.5-point difference [1.5, 11.5] (Experiment 2). This matches the total reasoning allowance, although not exact FLOPs.

The probes add 5,120 reasoning tokens, approximately USD 0.018 and 33 seconds of modeled latency per item. At near-equal accuracy, fixed depth achieves 70.5% at USD 0.0426, versus 70.0% at USD 0.0574 for the allocator, which costs approximately 35% more. At the shared 16,384-token ceiling, the allocator is cheaper in dollars but less accurate. A fresh 21,504-token baseline also favors depth by 4.0 points [0.0, 8.1], although that comparison partly mixes providers and its interval touches zero.

Your overhead question therefore has a substantive consequence: the allocator does not establish net savings over fixed depth. We will state this explicitly. Its supporting role is to test prediction of outcomes under candidate allocations. The additive predictor, including probe and provider features, improves held-out Brier score from 0.202 to 0.179 over a population-only fit (§E.2); better prediction alone does not establish a better routing policy.

We have now accounted for the probes and reported the requested common-budget comparison, including the result that favors the simpler baseline. Does this resolve your overhead concern? If not, please specify what remains missing from that comparison.

Please, let us know if we addressed your concerns and questions or we could do anything else to further strenghten the manuscript! Let us know also whether you have any remaining or new dubts! We're very willing to work with you towards strengthening the paper!

Best,

The authors

Reviewer_C94HOpenReview ↗

Response to author's answer

Thank you for the additional experiments. The new results strengthen the paper's claims regarding cross-model generalization and benchmark transfer. However, my main concern remains: under a matched total token budget, the allocator underperforms fixed-depth inference (64.0% vs. 70.5%), and it does not demonstrate a clear cost advantage at comparable accuracy. Thus, while the results support the predictability of inference-configuration performance, they do not yet establish the practical benefit of adaptive compute allocation.

Hello! Thanks a lot for engaging with our work!

We are not proposing a quantitative model for dynamic allocation. In fact, it's just throwing logistic regression at the problem :) We're characterizing:

regarding the different axis of test-time compute scaling, which direction gives the most reward and when?

In particular, we do not make a claim of proposing SoTA dynamic allocator. We're happy that you liked the fact that that dynamic allocator does a great job (us too!) but we didn't write that proposing that allocator was our contribution. We brought up the dynamic allocator to further corroborate our finding, as the title of the section where we introduce it claims: "Simple Allocation Learning Supports the Depth Finding". We're in fact happy to demote it if you think the dynamic allocation is confusing there.

Our claim is that deep reasoning is typically preferable over any other allocation of compute in multi-agent settings. We showed this varying per-agent compute budget and structure of the tasks (Figure 12). The fixed-depth inference at equivalent budget further supports this by demonstrating that error-rates in the allocator stem from not prioritizing depth!

However, my main concern remains: under a matched total token budget, the allocator underperforms fixed-depth inference (64.0% vs. 70.5%), and it does not demonstrate a clear cost advantage at comparable accuracy. [...] they do not yet establish the practical benefit of adaptive compute allocation.

The allocator is simply a supporting feature to further quantitatively demonstrate that allocating inference-time compute to deeper reasoning, given a known baseline sampling success rate, is modelably preferable across most settings.

Thanks a lot again for engaging. Thinking actionably about what I can do about it (please correct me if I'm wrong): I guess this means that we can do some work at rephrasing and making clearer Section 5 and Section 6. However, I believe the intro was already fine, except, probably I'd put Figure 12 in page 2. What do you think about it? Any other comment?

Again, thanks a lot for engaging, best of luck with your own submissions :)

Best,

The authors

Reviewer_8pbGOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper discusses the issue of allocating inference-time compute for LLM-based agentic systems under a restricted budget. The authors are motivated by conflicting findings in the literature on the matter, where different researchers recommend that one should scale in depth (e.g., with more thinking tokens), scale in width (e.g., with more agents), use debate (agents communicating with each other) or replace debate with voting. Based on their descriptive analysis (i.e., running different combinations of allocation strategies and reporting findings), the authors provide practical suggestions about resource allocation. The claimed contributions of the paper are the following:

  1. An evaluation framework for different setups, consisting of three main parts: measuring cost (= agents * rounds * tokens per call), different regimes (problem difficulty w.r.t. a model's capability for that problem) based on the success rate of 25 short queries, and fine-grained description of error provenance, i.e., whether the error comes from generation (the correct answer was never generated), preservation (the correct answer was generated at some point, but it didn't make it to the final stages) or selection (the correct answer was preserved at the final stage, but was not selected as the final answer).
  2. Empirical findings from the analysis of the evaluation framework. The authors found that for more difficult problems, it is better to use more thinking tokens in one LLM query than fewer thinking tokens with multiple LLM queries (under the same total budget). By contrast, for easier problems, the authors found that using fewer thinking tokens, with multiple LLM queries with an answer derived by majority vote, performs as well as one long LLM query, at lower cost. Regarding debate, they found that debate underperforms a single long LLM query (with more thinking tokens) when put on the same total budget.
  3. They learn a logistic regression allocator model that tries to predict the best strategy beforehand. They run 5 short queries per problem to get the features, which are then fed to the model. The model mostly learned to recommend a single long query.
Critical strengths
  • The paper tackles a problem that has practical importance. The major contribution, in my opinion, is defining the evaluation framework. Consequently, this paper provides a basis for subsequent work, which could find a reliable way to allocate resources with a minimal number of short diagnostic queries. This can be very useful, especially for complex tasks.
  • Using the evaluation framework, the authors provide findings that are grounded in a more systematic framing than prior work (e.g., studies recommending more agents, or replacing debate with voting). In particular, the authors force the agentic systems to use their entire budgets, making the configurations directly comparable. With their approach, they shed light on conflicting claims in prior work.
Critical weaknesses
  • I found the paper quite difficult to follow. I had to re-read some things a few times, or even to go back and forth between sections, in order to understand what the authors wanted to say. The arguments and descriptions seem to be scattered across different sections (e.g., the term "regimes" appears in the title and intro without even an intuitive explanation, yet it is first defined in Section 3, Preliminaries); the authors seem to use several terms for one concept (e.g., attempt, draw and probe), etc.
  • I think that one important baseline is missing from the experiments: a default mode, in which the model stops when it decides to, rather than being forced to exhaust its budget. Without this baseline, it’s unclear whether forcing full budget use helps or hurts, and whether the findings carry over to how these systems are run in practice.
Questions for authors

No further questions.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8pbG

Thank you for recognizing the value of the evaluation framework and its potential to support future work. Let us know if below we address all your concerns and questions conclusively! We are very willing to further work with you to clear up any further question! You raised two concerns: the presentation was difficult to follow, and the experiments lacked a natural-stopping baseline. We address each directly below.

1. Organization, definitions and terminology. Your examples identify where the explanation needs to improve. We will make four changes:

  • Open with one running example: should 16,384 reasoning tokens be spent on one long attempt or four attempts of 4,096 tokens each? We will present the comparison protocol, findings, error accounting and allocator in that order.
  • Define a regime in the Introduction as a group of problems on which a particular model has similar observed success under the same short reasoning allowance. For example, “no observed success” means zero correct answers in 25 reference attempts; it does not mean zero success probability or an unsolvable problem.
  • Use “attempt” for a generated solution and reserve “probe” for an attempt used to inform allocation. The 25 reference attempts define retrospective groups using correctness labels. The five deployment probes provide routing features without access to their correctness labels.
  • Simplify the notation and state the practical message in an introductory box: “In the tested harder regimes at our main budget, one deeper call is more accurate than splitting the budget across the evaluated agents and rounds. When short attempts are already reliable, inexpensive voting can suffice. Any allocator used on new items must also account for its decision cost.”

These changes address the scattered presentation, late regime definition and inconsistent terminology you identified. Do they address your concerns about the proposed organization? If another definition or transition needs attention, please point us to it so we can revise it as well.

2. Natural stopping. You ask whether forcing full spending helps or hurts, and whether the depth advantage survives when the model stops on its own. We have now run that comparison on Qwen3.8-27B using 96 problems and two repeats per item (Experiment 1). At the same 16,384-token reasoning ceiling:

ProtocolDepth minus four-way voting, percentage points [95% CI]
Forced reasoning, then a separate answer+10.9 [5.2, 17.2]
Natural stopping, then a separate answer+14.1 [6.8, 21.9]

Depth remains more accurate without forced spending. Forcing improved voting by 6.8 points [1.6, 12.0], while its effect on depth was inconclusive. The difference between the two depth–voting gaps was +3.1 points [−4.2, 9.9], so we do not claim that natural stopping significantly increases the advantage.

Under natural stopping, depth used 10,744 reasoning tokens and cost USD 0.0268 per item, versus 13,558 tokens and USD 0.0367 for voting. It was therefore more accurate while using approximately 21% fewer reasoning tokens and costing 27% less. The common ceiling did not imply equal expenditure.

We originally required full spending to separate allocation from stopping behavior. We also tested a single-call protocol with one cap shared between reasoning and answering. Here, 64% of voting calls exhausted their cap before answering. Since the total caps also differ between depth and voting, we will report this separately as an answer-truncation failure, not an equal-budget test of reasoning quality.

The requested natural-stopping comparison is therefore complete and supports the depth result, we believe. Let us know if you have reservations! Does this resolve your concern that the finding might depend on forced spending?

Please let us know of concerns and questions, if any, remain! We would be happy to address them as fast as possible. Thank you again :)

Hello, thank again for the thoughtful review! Just one last message to say that in cas you had any actionable comment or advice for us to improve the paper, we're happy to work through it fast.

Regradless, let us know if you have any further comment or question!

Good luck with your submissions :)

Best,

The authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Rebuttal overview: completed experiments and main conclusions

Thank you to the reviewers for their careful reading and thoughtful questions. We have completed the following experiments and analyses in response. The table summarizes what we tested, why we tested it, and which concerns each study addresses.

StudyWhy we ran itMost relevant reviewsMain finding
1. Stopping protocols (Qwen3.8-27B)Test whether forced spending drives the result8pbGDepth leads voting by 14.1 pp with natural stopping and 10.9 pp with forced spending.
2. Full probe cost (200 held-out problems)Compare accuracy and cost after charging all five probesC94H, j9dPAt a 16,384-token all-in ceiling: 70.5% fixed depth vs 64.0% allocator.
3. Gemini protocolsTest a proprietary family and shared reasoning/answer capsC94H, j9dP; also 8pbGWith room to answer, depth leads voting by 5.7 pp; shared-cap comparisons are inconclusive.
3b. Three additional families (DeepSeek, Claude, GPT)Test generalization beyond QwenC94H, j9dPDepth leads DeepSeek voting/debate and Claude debate; Claude voting and both GPT contrasts are inconclusive.
3c. Budget sweep (DeepSeek and Claude, 4,096–65,536 tokens)Test whether the advantage depends on budgetC94H, j9dPDepth leads by 19.8–25.0 pp at 4,096 tokens; gaps narrow as budgets increase.
4. Model-size comparison (Qwen3-8B/32B)Test different parameter scales with natural stoppingC94H, j9dPDepth leads voting/debate by 10.9–14.6 pp.
5. Compute accounting (dense Qwen3)Include prefill, decoding, caching and latencyj9dP; also C94HWidth uses 1.68–2.93× estimated FLOPs, or 1.57–2.68× with prefix caching; parallel width is faster.
6. Benchmark transfer (four held-out-benchmark fits)Test target-data requirements and recalibration costC94HWithout target training labels, aggregate accuracy matches the full fit; target labels improve calibration.

Study details. “pp” means percentage points. The added model and stopping comparisons use 96 problems from four benchmarks, with two repeats per item. Experiment 2 uses a separate 200-problem holdout and combines stored results with a fresh 21,504-token baseline. Experiment 3c reuses the 16,384-token runs; Experiments 5–6 are compute and transfer analyses. We distinguish nominal allowances and provider effort settings from actual expenditure. The reported 95% intervals are pointwise, and comparisons that share runs are not independent replications.

Across these tests, depth remains a strong baseline across several models and stopping protocols, particularly under tight budgets. Charging the routing probes also favors fixed depth. Although the fitted predictors help predict outcomes under candidate allocations, their predictions do not establish net savings. A more elaborate allocator must therefore justify the cost of the information it uses.

We will add these results to the revision and reorganize the paper around a running allocation example. We will define model-relative regimes when they first appear, distinguish reference attempts from deployment probes, and state the assumptions of the illustrative models separately from the exact error accounting. The revised results will include both positive and inconclusive comparisons.

In the replies below, we explain how each comparison answers the concern you raised. Please let us know whether that concern is resolved :) If it is not, please let us know the specific claim, missing comparison or uncertainty that remains, we are really keen to addressing then directly and quickly! Thank you again for helping us strengthen the manuscript :) and good luck with your own submissions!

The authors

Thank you to all of the reviewers for your valuable feedback. We have taken it into account and made some updates to the paper accordingly, including more clearly highlighting our core claims in section 5 and moving our a key diagram demonstrating our core claim from the appendix to earlier in our paper (now figure 1, in which we clearly demonstrate the identified regimes and the performance of each allocation strategy across the regimes).

It appears that a key point of confusion among some of the reviewers was the idea that our learned dynamic allocation rule was a core contribution of the paper. It was not, and we apologize for any lack of clarity that may have indicated this.

The core contribution of the paper is understanding how scaling compute allocation across "width" versus "depth" affects agentic system performance. Situated within the current literature on multi-agent systems for amplifying agentic system correctness, our key contribution is simply demonstrating that with a fixed aggregate compute budget, in most settings (for which we clearly define the settings as "regimes" in section 5.1 of our paper), particularly the"harder" regimes simply allocating compute budget to a single agent with deeper reasoning is superior to any strategies attempting to use multiple agents and/or multiple rounds with debate, vote, or revision.

The learned dynamic allocator was simply a measure to observe whether the relationship that we observed regarding depth is modelable in a quantifiable way. Moreover, comparison to a fixed depth baseline ("deeper reasoning") simply reinforces our observation that in most of the regimes deeper reasoning beats other forms of distributing compute in more intricate manners.

Once again, we thank all of the reviewers for their valuable feedback. We have made some edits to the submitted paper, but will continue to make improvements to how we frame and communicate our results to make our findings more clear in line with your feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.