acceptodds
Withdrawn from ICLR 2027

Compute-Optimal Stopping for Multi-LLM Deliberation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-LLM systems can spend substantially different amounts of inference compute on the same query, making adaptive stopping essential for balancing answer quality and computational cost. We present Dwell, a cost-sensitive stopping framework for selecting among fixed-depth multi-LLM deliberation policies. Under diminishing marginal accuracy per unit cost on the efficient frontier, we show that the cheapest utility-maximizing endpoint follows a threshold rule and that dominated depths can be excluded as stopping endpoints. In a controlled five-seed simulation, the selected depth-four endpoint achieves 0.950 ± 0.011 accuracy at a normalized cost of 7.1, compared with 0.964 ± 0.010 accuracy at a cost of 19.2 for mixture-of-agents. The resulting operating point is 2.7× cheaper while sacrificing only 1.4 accuracy points. A lightweight confidence model obtains an expected calibration error of 0.102, and our conditional headroom bound connects calibrated confidence to the value of further deliberation. A held-out 60-item ARC-Challenge study with three API-served workers further characterizes how the preferred endpoint changes with the cost coefficient λ. Together, these results establish a principled mechanism for selecting deliberation depth under explicit accuracy–cost trade-offs while separating controlled evidence from the preliminary live-model evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-LLM Deliberation, Test-Time Compute, Adaptive Inference, Optimal Stopping

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_whScOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes DWELL, a cost-sensitive framework for selecting among predefined, fixed-depth multi-LLM deliberation policies by maximizing expected accuracy minus a weighted inference cost. Under diminishing marginal returns along the accuracy–cost frontier, it derives a threshold rule for selecting the cheapest utility-maximizing endpoint. It also establishes an upper bound on the expected benefit of further deliberation, assuming state-conditionally calibrated confidence.

Validation consists of a five-seed controlled simulation and a preliminary ARC-Challenge evaluation with 60 held-out questions and three API-served models. The simulation reports a selected endpoint with 2.7× lower normalized cost than mixture-of-agents at a 1.4-percentage-point accuracy loss.

Critical strengths

The proposed problem is important in practice.

Critical weaknesses

The paper is very poorly written and difficult to follow. It is full of undefined terms and uninformative, defensive statements.

Questions for authors

None.

Reviewer_2ZDdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DWELL, a method for choosing a deliberation depth t from a fixed family of multi-LLM policies. Policy t takes a plurality vote over the first t workers in a fixed order, and DWELL picks the t that maximizes accuracy minus a cost penalty. Theorem 1 says that when the accuracy–cost frontier has diminishing marginal returns, the cheapest maximizer is given by a threshold rule on the slopes between efficient endpoints. Experimental evidence is a five-seed synthetic simulation, where depth 4 scores 0.950 at cost 7.1 versus mixture-of-agents at 0.964 and cost 19.2, plus a single 60-item ARC-Challenge split with three API-served models.

Critical strengths
  1. The scope statements are honest and correct. The paper distinguishes distribution-level endpoint selection from per-query control. It notes that a dominated depth is skipped only as a stopping point and still has to be computed on the way to deeper endpoints. It also states that marginal ECE does not imply the state-conditional calibration Proposition 1 needs. This matters to my assessment because the claims, read literally, are not overstated. The problem is that, read literally, they are small. The paper's own framing therefore limits how much contribution I can credit.

  2. Accounting for all protocols on one cost scale exposes dominated protocols. On ARC-Challenge, best-single matches self-consistency at one third of the cost, and the panel vote matches debate at half the cost. This is a useful caution for multi-agent work, which often reports accuracy without cost. It counts in the paper's favor, but with one split of 60 items it is anecdotal.

Critical weaknesses
  1. The contributions are close to definitional and do not justify a paper. Choosing t from five measured (Acc, Cost) pairs is an argmax over five numbers. Theorem 1 characterizes the result as a threshold rule, which is the standard argument for the upper concave hull, and its proof is six lines. The rule also needs Assumption 1, while plain enumeration works without it. Proposition 1 ignores that continuing can also lower the score, the bound is loose enough to be uninformative. This weighs most heavily on my assessment: a stronger experimental section would not supply a technical contribution the paper lacks.

  2. The headline result is not attributable to DWELL, and the evidence comes from a simulator that is not described. The abstract's "2.7× cheaper while sacrificing only 1.4 accuracy points" compares the depth-4 plurality vote with simulated mixture-of-agents. That is a comparison between a fixed-depth vote and another protocol; DWELL's selection step contributes nothing to it. The depth was also chosen as an oracle on the same distribution-level estimates. The simulator's design determines the result, and it is not described. On real models, the only evidence is the six-question ARC difference between baselines that are not implemented as named.

Questions for authors
  1. Baseline fidelity. On ARC-Challenge, does "mixture-of-agents" call an aggregator LLM, or is it plurality voting over the three workers? The costs (2.4 = sum of the three workers) suggest the latter. If so, can the baseline names in Tables 1 and 3 be justified?

  2. Simulator specification. How are worker skills, correlated errors, debate, and mixture-of-agents generated in the five-seed simulation? Without this, Table 1 and the headline numbers cannot be evaluated or reproduced.

  3. In-sample selection. Are the DWELL-Oracle endpoints in Table 1 chosen on the same data used to report their accuracy? If they are, what are the held-out results?

  4. Worker accuracy on ARC-Challenge. Best-single is the cost-0.6 worker (Llama-3.1-8B) at 0.783, which means GPT-4o-mini scored no higher on ARC-Challenge. Is that an answer-extraction failure, given the 256-token cap and multiple-choice letter parsing? This affects every ARC number in the paper.

Reviewer_kTPPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DWELL, a framework for choosing where to stop in a multi-LLM deliberation pipeline under an explicit accuracy–cost objective U_λ(t) = A(t) − λC(t). Theorem 1 shows that under diminishing marginal returns along the efficient frontier (ρ₁ ≥ … ≥ ρ_m > 0), the cheapest utility-maximizing endpoint is the last frontier point reached through steps with ρ_ℓ > λ, and that dominated depths can be excluded as stopping choices. Proposition 1 gives a headroom bound E[Y_u − Y_t | Z_t] ≤ 1 − p_t(Z_t) under state-conditional calibration. Validation is a five-seed synthetic sweep over five fixed depths where the selected depth-four endpoint reaches 0.950 accuracy at cost 7.1 versus 0.964 at 19.2 for mixture-of-agents, plus a 60-item ARC-Challenge split with three API workers and a logistic confidence model at ECE 0.102.

Critical strengths

The paper is scrupulously honest about what it does not establish — that oracle selection adds no new operating points, that the controller is unevaluated, and that aggregate ECE does not establish Proposition 1’s premise. The λ-sweep framing of endpoint choice is a reasonable way to compare protocols at a common exchange rate.

Critical weaknesses

See questions.

Questions for authors

In ARC-Challenge (Table 3), the endpoints are distinct protocols — best-single, self-consistency, MoA, debate — not prefixes of a common worker order as Π_fix requires (Sec 3.1). Theorem 1’s traversal semantics do not apply there. In what sense does this study validate the fixed-depth framework rather than simply tabulating four protocols?

Pareto efficiency does not imply Assumption 1: efficient points can have increasing marginal rates (e.g. A = .5, .6, .9 at costs 1, 2, 3). In your own sweep A(t) is not even monotone — depth two scores 0.783 at cost 3.1 versus 0.792 at cost 1.0. Given |Π_fix| = 5, what does the threshold rule provide over directly enumerating U_λ(t), particularly when the assumption fails?

Proposition 1 follows from Y_u ≤ 1 alone and uses no property of the continuation. At λ = 0.01 and ΔC = 2.4 (the 3→4 step), the stopping condition 1 − p_t ≤ λΔC requires p_t ≥ 0.976. How often would this fire, and does it ever recommend stopping earlier than the frontier rule? Sec 3.3 states that oracle selection uses A(t) and C(t) independently of p̂_t, and Sec 3.4 states the controller is not trained or evaluated.

What claim does the ECE 0.102 result support, given the confidence model enters no reported decision?

Reviewer_gkw4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes DWELL, a cost-sensitive stopping framework for selecting among fixed-depth multi-LLM deliberation policies. This paper also formulates cost-sensitive endpoint selection for a predefined distribution-level fixed-depth policy family and proves a cheapest-optimum threshold characterization under diminishing frontier returns.

Critical strengths
  1. This paper proposes a cost-sensitive stopping framework for selecting among fixed-depth multi-LLM deliberation policies.
  2. Under diminishing marginal accuracy per unit cost on the efficient frontier, this paper shows that the cheapest utility-maximizing endpoint follows a threshold rule and that dominated depths can be excluded as stopping endpoints.
Critical weaknesses
  1. The writing is very unclear. In the introduction, it does not clearly introduce the background or the problem. Hence, I don't know what the motivation for this paper is. In the related work, the listed literature is poorly categorized; what is the connection between the related work and this paper? In the method section (i.e., Section 3), it is also confusing. What is the meaning of distribution-level fixed-depth policies? The notations and terminology are not fully explained; hence, I can not understand how the method actually works. In the theory section (i.e., Section 4), in Definition 1 (Efficient stopping endpoints), what is the meaning of endpoints? And for Assumption 1 (Diminishing frontier returns), why does this paper make this assumption? Does it really hold in the experiment? And for Theorem 1 (Cheapest threshold optimum), what is the application of this theorem? The whole theory part can not be understood well.

  2. The experiment is not solid. The setting is not clear; for example, there is one sentence in the paper: "The controlled simulation uses heterogeneous workers whose family-conditioned skill profiles create complementary candidate panels and whose errors include a shared correlation component." What is the meaning of heterogeneous workers?

Questions for authors

How does the DWELL method actually work? What are the inputs and outputs of this method? What are the Cost-aware Controller, Black-box worker pool, and Learned Synthesizer?

What is the main idea of this paper's theory section?

In the experiment, what does each baseline mean? And why use only one dataset?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.