acceptodds
Under review as a conference paper at ICLR 2027

When Should LLM Agents Talk? Measuring Coordination Value and Auditing Learned Routers

Abstract

After a language-model agent produces an initial answer, the system must decide whether further coordination is worth the inference cost. Even if an initial answer is assessed as incorrect, coordination may not improve it enough to justify its cost. We define the policy-relative Causal Value of Coordination (CVC) as the expected difference in cost-adjusted utility between a coordination action and a feasible alternative. Both start from the same history and follow the same fixed continuation policy. With the incumbent policy as the reference, a performance-difference identity links exact CVC values to end-to-end policy improvement. We estimate CVC through paired counterfactual replay and train ridge and neural predictors to construct candidate routing policies. Frozen candidates are assessed through full-policy comparisons and audits. Our main evaluation uses 2,000 MMLU-Pro questions across two backbones, complemented by supplementary GSM8K and MMLU-Pro experiments. Confidence predicts initial errors, but the prespecified router comparisons do not establish a utility gain. A diagnostic with access to initial correctness yields larger gains, highlighting the gap between error prediction and effective routing. A fixed early-stopping policy reduces voting cost without changing answers; observable-failure recovery yields modest secondary gains. Learned-policy lower bounds remain negative, and no candidate passes the supplementary independent audits. These findings highlight that identifying beneficial coordination, learning effective routing policies, and certifying end-to-end improvements are distinct challenges under the tested conditions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Multi-agent systems, Agent communication, Adaptive inference, Counterfactual evaluation, Policy evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_G25bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper defines the policy-relative Causal Value of Coordination (CVC). CVC compares the cost-adjusted utility of a coordination action with that of a feasible alternative. The comparison uses the same history and a fixed continuation policy. The authors use paired counterfactual replay to estimate differences in utility and cost. They train routers using text, confidence, and frozen embeddings. They then assess gains through full-policy evaluation on held-out data and audits based on lower confidence bounds. The main experiments evaluate Qwen3-8B and Gemma-3-12B-it on the same 2,000 MMLU-Pro questions. Additional experiments include GSM8K. The results show that confidence can predict whether the initial answer is incorrect.

Critical strengths
  1. The paper clearly distinguishes initial error prediction from assessing the value of coordination.
  2. The measurement and evaluation procedures are well structured, with attention to selection bias and the unit of analysis.
  3. The negative empirical findings are informative and include a detailed breakdown of outcomes.
Critical weaknesses
  1. The audit guarantees rely on an independent and identically distributed (i.i.d.) assumption. However, the main evaluation uses category-stratified sampling without replacement.
  2. Most evidence comes from question-answering tasks with fixed prompts and aggregation rules. The decision is whether to consult a peer after the initial answer. The appendix expands the action set and includes two-node routing. However, it does not provide systematic validation with heterogeneous agents, complex tool-use tasks, or long-horizon, multi-round coordination.
  3. The main cost metric does not capture the full online routing overhead.
Questions for authors
  1. How should confidence-bound coverage and detectable gains be interpreted under the current sampling design? Please explain how the i.i.d. approximation affects confidence-bound coverage under stratified sampling without replacement. Is there any validation or derivation that matches the actual sampling design? Given the current sample size and error budget, how large must a utility gain be for the audit to detect it reliably? This information will affect my interpretation of the audit guarantees and the fact that no candidate passed.
  2. Which conclusions generalize beyond the current coordination protocol? The paper already includes an expanded action set and two-node experiments. Please clarify which findings mainly depend on the current peer-generation process, aggregation rules, and task settings. Which findings are supported across settings? Are results available for heterogeneous peers or other coordination protocols under the same evaluation procedure? This will affect my assessment of the generalizability of the empirical findings and the scope of the paper’s contribution.
  3. Do the main policy comparisons still hold after including feature extraction and scoring costs? Please explain how including these additional costs changes each router’s utility relative to immediate stopping (Stop) in the main evaluation. The costs should include frozen embedding extraction, teacher-forced scoring, and input tokens. Are timing or cost measurements available for the full online execution path? These results will affect my assessment of cost-adjusted utility and the practical implications for deployment.
3 comments
AuthorsOpenReview ↗

Response to Reviewer G25b (1/3)

We sincerely thank Reviewer G25b for the precise and constructive questions. Each of them led to a concrete analysis and to a change in the paper. Tables R2, R3, R5, and R6 are in the general response.

[W1 & Q1] The i.i.d. assumption under category-stratified sampling without replacement

Thank you for pointing this out. The submission left the matter open, and we have now worked it out for the actual design, in which 2,000 questions are drawn without replacement from the remaining pool of 3,131 with category-proportional quotas, and the target is the pool mean.

The Hoeffding bounds remain valid, up to a quota-rounding term. Hoeffding (1963, Thm. 4) shows that, for any continuous convex ff, the expectation of ff of a sum sampled without replacement is at most its expectation under sampling with replacement. The bounds on the moment-generating function therefore carry over within each category. Since the categories are sampled independently, these bounds multiply, and the i.i.d. Hoeffding inequality holds for the stratified sample mean around its expectation. With integer quotas, this expectation differs from the pool mean by at most 0.7% of the largest gap between a category mean and the overall mean. The post-stratified estimate removes this term: the same argument applies to it with ∑hWh2/nh\sum_h W_h^2/n_h in place of 1/n1/n, where WhW_h is the pool share of category hh, and no point estimate changes by more than .00002. The same argument covers Prop. 2 whenever audit questions are drawn without replacement from a finite pool. Applying Serfling's finite-population refinement within each category (Bardenet and Maillard, 2015, Prop. 2.2) reduces the penalty from .062 to about .037, because 64% of the pool is sampled.

The empirical-Bernstein bounds were an approximation, since the Maurer–Pontil bound is proved for i.i.d. samples. We computed its analog for simple random sampling without replacement (Bardenet and Maillard, 2015, Thm. 4.3) from the per-question differences. For the four audited routers, its penalties are .030–.033, against .019–.025 for the bound in the paper, because its range term has a larger constant. It does not cover stratification, so we report it for reference, treat the Hoeffding bound as the design-valid one, and label the Maurer–Pontil values as approximate.

We also computed design-based intervals from the per-question differences, with category weights from the pool and the stratified finite-population standard error.

Contrast on the main 2,000 questionsGain and bootstrap interval (paper)Design-based intervalEB lower bound (paper)EB–Serfling lower bound
Qwen, selected router − Stop+.0034 [−.0060, .0128][−.00223, .00905]−.021−.030
Gemma, selected router − Stop+.0012 [−.0079, .0104][−.00427, .00665]−.023−.032
Qwen, text-feature router − Stop+.0036 [−.0020, .0095][.00011, .00715]−.016−.026
Gemma, text-feature router − Stop+.0068 [.0004, .0137][.00286, .01079]−.014−.024
Qwen, confidence classifier − Stop+.0090 [−.0026, .0207][.00204, .01592]−.019−.027
Gemma, confidence classifier − Stop+.0056 [−.0063, .0173][−.00145, .01272]−.023−.030

The design-based intervals are about 40% narrower than the bootstrap intervals, which treat the sample as i.i.d. and are therefore conservative for the pool mean. Under the design-based reading, a few contrasts exclude zero, including the confidence classifier against Stop on Qwen. These intervals condition on the realized responses and are not adjusted for multiplicity, so we report them as a sensitivity analysis and do not base conclusions on them. Every lower bound stays negative under every version, and no audit decision depends on the approximation. Table R7 in App. A.5 gives all primary and secondary contrasts.

AuthorsOpenReview ↗

Response to Reviewer G25b (2/3)

To check the sampling argument independently of the data, we also ran a simulation that matches the design. We drew 2,000 questions without replacement under the category quotas 20,000 times from a synthetic pool of 3,131 questions across 14 categories. The pool is calibrated to the selected router (coordination on 45.4% of the questions, per-question standard deviation of .21), and the spread of repair and harm rates across categories is assumed. No bound exceeded the pool mean in any draw. The reported 95% interval covered the pool mean in 99.9% of draws, and a design-based interval covered it in 94.7–95.3% of draws.

On the detectable gain, the reported intervals have a standard error of about .0048 and reach 80% power near a gain of .013, or .019 after correction over the twelve primary contrasts. For the four audited routers, the penalties are .019–.025 for empirical Bernstein and .037 for the design-matched Hoeffding bound, and the penalty of the supplementary audit is .110 (n=1,000n = 1{,}000, K=3K = 3). The statement that no candidate passed should therefore be read as "not certified at this resolution", which is how Sec. 4 words a declined audit. The held-out tests added during the discussion period (Table R5) give a direct illustration. A confidence-based trigger gains about .012 to .016 over Stop on 3,131 further questions, which is at the limit of what the paired intervals could detect on the main sample and below what any of the bounds could certify.

[W2 & Q2] What generalizes beyond the current coordination protocol

Thank you for this question. Table R3 lists each finding with the settings in which it was observed, and we summarize it here.

Three findings are supported across the settings that we tested. The harm rate of consulting rises with predicted error on the main sample and on both held-out sets. EMS3 keeps the answers of Vote while saving cost, which is structural and holds on three backbones. The learned value router is not established over its reference, for the Stop, Vote and Review-then-Stop references and for binary, four-action, post-peer and two-node decisions.

Other findings depend on the peer, the aggregation rule, or the task. These are the value of always coordinating against Stop, the failure-trigger gain (a recovery under the 2,048-token cap), and the sensitivity to the scoring rule (on Qwen, +0.2 points under normalized scoring against +26.6 under strict extraction). For heterogeneous peers, we added three analyses during the discussion period (response to Reviewer YGUP, Q1, and Table R9). Because both backbones answered the same 2,000 questions, cross-backbone voting can be scored from cached responses. A Gemma answer voted with two Qwen peers gains +4.70 points [3.10, 6.40], while a Qwen answer voted with two Gemma peers gains −0.35 [−1.95, 1.20]. With 2,000 new aggregator calls per direction, aggregation with a peer from the other backbone gains +2.50 [1.10, 3.90] for a Qwen answer and +3.95 [2.35, 5.60] for a Gemma answer, against +1.55 [0.20, 2.90] and +1.10 [−0.30, 2.50] with a same-backbone peer. In an exploratory run with a Qwen3-32B peer, the repair rate roughly doubles, but the aggregate is not better than the stronger model answering alone. The benefit of a heterogeneous peer therefore follows its relative strength.

Other protocols under the same procedure are already in the paper, namely self-review, voting, early stopping, aggregation after a peer, a four-action menu, and two-node review-then-coordinate on GSM8K.

AuthorsOpenReview ↗

Response to Reviewer G25b (3/3)

[W3 & Q3] Policy comparisons once feature extraction and scoring costs are included

Thank you for this question. The primary objective counts output tokens only (App. A.4). We have now measured the overhead of the full online path on one H100 GPU with vLLM 0.11.0, using 200 questions for the scoring pass and the encoder, and 64 questions for generation.

Measured per questionQwen3-8BGemma-3-12B-it
Initial answer: prompt tokens / output tokens262 / 535257 / 609
Teacher-forced scoring pass: tokens / seconds797 / 0.058866 / 0.104
Encoder pass (Qwen3-Embedding-0.6B): tokens / seconds722 / 0.035790 / 0.038
Generating one initial answer, single stream: seconds5.5911.24

The overhead is paid on every question, before the decision, and only by policies that compute features. Stop, Coordination, Vote, EMS3, the text-feature router (CPU features only), and the failure trigger pay nothing. The confidence classifier and the break-even trigger need the scoring pass, and the selected router needs the scoring pass and the encoder. The table below gives the gain over Stop on the main 2,000 questions under three charges.

PolicyOutput tokens only (paper)Overhead charged by measured timeOverhead charged as output tokens
Qwen, selected router+.0034 [−.0060, .0128]+.0033 [−.0061, .0127]−.0118 [−.0212, −.0025]
Gemma, selected router+.0012 [−.0079, .0104]+.0011 [−.0080, .0103]−.0154 [−.0245, −.0061]
Qwen, confidence classifier+.0090 [−.0026, .0207]+.0089 [−.0026, .0206]+.0010 [−.0106, .0127]
Gemma, confidence classifier+.0056 [−.0063, .0173]+.0056 [−.0064, .0172]−.0030 [−.0149, .0086]

In wall time, the overhead is small. The scoring pass and the encoder together take about 0.09 s on Qwen and 0.14 s on Gemma, against 5.6 s and 11.2 s to generate one initial answer, which corresponds to about 8 output tokens. Charged by measured time, every gain changes by .0001 or less. In token count, the overhead is not small, since the scoring pass alone re-reads more tokens than the initial answer generates. If each of these tokens were priced like a generated token, the selected router would fall below Stop on both backbones. The two charges bracket the cost: a forward pass over existing tokens is cheaper than generating them, but it is not free.

The same accounting applies to the held-out tests. Charged by measured time, the gain of the break-even trigger is unchanged. Under the upper-bound charge it becomes +.0125 [−.0030, .0287] and −.0036 [−.0191, .0122] on the prospective set and +.0059 [−.0052, .0173] and +.0078 [−.0044, .0204] on the confirmation set. This charge is deliberately pessimistic, since the scoring pass takes 0.058 s and 0.104 s per question, against 5.6 s and 11.2 s for generating one initial answer. On Qwen in the prospective test, the gain remains positive as long as a scoring-pass token costs less than 2.6 output tokens. We report these results in App. A.4 (Table R8) and with the held-out tests in App. A.7.

We hope this resolves questions about the audit guarantees, the scope of the findings, and cost accounting. We would be glad to answer any further questions.

Reviewer_6BrROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether additional coordination is worthwhile after an LLM produces an initial answer. It defines policy relative Causal Value of Coordination, estimates it through paired counterfactual replay, and trains routing policies that are evaluated and audited after selection. Experiments primarily use MMLU-Pro with Qwen3-8B and Gemma-3-12B-it, supplemented by GSM8K. Confidence predicts initial errors, but the prespecified learned-router comparisons do not establish reliable utility gains, and no candidate passes the supplementary independent audits.

Critical strengths

--- Relevant problem and useful evaluation. Distinguishing initial error prediction from the expected benefit of coordination is useful. Separating policy selection from held-out evaluation strengthens the credibility of the reported results.

--- Transparent negative findings. The repair/harm analyses and explicit reporting of uncertainty provide useful evidence about the tested coordination policies.

Critical weaknesses

--- Limited incremental insight. Several central observations have substantial precedent: voting explains much of debate’s benefit (Debate or Vote); communication can corrupt correct answers (Talk Isn’t Always Cheap); coordination benefits depend on capability, task structure, and overhead (Towards a Science of Scaling Agent Systems); and available routing headroom can be difficult to recover from observable signals (Statistical Scouting). The replay and audit procedure is a useful contribution, but the paper does not sufficiently establish what new understanding it provides beyond these findings.

--- The empirical evidence has limited scope and diagnostic resolution. The main experiments involve one routing decision, while the sequential extension contains only two. The important domain long-horizon coordination is untested. The main backbones are Qwen3-8B and Gemma-3-12B-it, with Qwen thinking disabled, so robustness to more recent reasoning configurations remains unclear. Furthermore, the supplementary MMLU-Pro audit penalty is approximately 0.110, compared with positive estimated gains of only 0.005–0.012.

Questions for authors

--- What substantive finding enabled by the replay and audit procedure goes beyond the existing evidence on harmful communication and difficult to exploit routing headroom?

--- Which findings are expected to depend on the short decision horizon and model configuration?

--- Given the reported audit resolution limitations, can the evaluation distinguish limitations of the learned routers from insufficient power to detect modest gains?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6BrR (1/2)

We sincerely thank Reviewer 6BrR for the review and for the four references, which we have added to Sec. 2. Below, we address each point with the results in the submission and with the experiments added during the discussion period. Tables R1 to R3, R5, and R6 are in the general response.

[W1 & Q1] What the replay and audit procedure shows beyond existing evidence on harmful communication and routing headroom

Thank you for this question. Our results agree with the cited studies at the population level. Consultation overturns 6.0% and 7.2% of correct answers, its net value depends on the backbone, and the learned routers gain little over their reference. Those studies compare protocols by accuracy. Paired replay adds the cost-adjusted outcome of both branches for every question, and we believe that three findings go beyond the existing evidence.

The first is where the harm sits relative to deployable signals, and what follows from it. Wynn et al. show that correct answers are abandoned in debate. Table R1 shows that this happens mostly in the low-confidence region that error triggers select. The harm rate among correct answers rises from about 1% to 20–25% across confidence bins, so that the error rate needed to break even rises from about 12% to 61–62%. Hu et al. report that vote entropy finds cases where debate is safe but not cases where it is useful. Our analysis gives a cost-adjusted account with signals that are available before any extra sample is drawn, and it leads to a rule that can be tested. In two prespecified held-out tests, a trigger built from this rule gains over Stop (+.0163 [.0071, .0257] and +.0124 [.0026, .0220] over the two sets together), and the gain is located in the least confident answers (Tables R5 and R6).

The second is a bracket on the headroom. Headroom is usually reported with an oracle that picks the best protocol per example from realized outcomes. On our data, the five-action version of that oracle shows +.079 and +.083. No policy can exceed it on the same replays, but it also selects on replay noise. The post hoc cross-replay selector chooses between Stop and Coordination with one replay and is scored on an independent one. Its expectation is a lower bound on the gain of the best history-measurable policy (Prop. 3), and it shows +.0115 [.0035, .0214] and +.0312 [.0173, .0471] in the supplementary evaluation (Table 19). Both intervals exclude zero, which indicates that headroom exists, and the two figures bracket the gain of the best policy. The lower bound requires independent replays from a shared history.

The third is what evidence a routing claim requires. Certifying a gain of +.01 takes about 92,000 questions under Hoeffding (App. C.4), whereas an answer-preserving change such as EMS3 is supported at 500 to 2,000. Choosing a router on evaluation outcomes produces +.0056 and +.0107 with intervals excluding zero, where the frozen candidate shows −.0004 and −.0001 (Table 19). The held-out tests add an example in the opposite direction. A gain of about .012 to .016, which the main sample of 2,000 questions could not resolve, was observed in two prespecified tests on 3,131 further questions.

AuthorsOpenReview ↗

Response to Reviewer 6BrR (2/2)

[W2 & Q2] Dependence on the short decision horizon and the model configuration

Thank you for raising the question of scope. During the discussion period we examined the model configuration, namely the reasoning mode (below) and the peer model (response to Reviewer YGUP, Q1). Table R3 lists each finding with the settings in which it was observed. The answer-preserving saving of EMS3 is structural for any majority-of-three vote and appears on all three backbones, so we expect it to be independent of horizon and configuration. The rise of the harm rate with predicted error, and the absence of an established gain for the learned value router, held in every setting that we tested, with magnitudes that depend on the configuration. The failure-trigger gain is configuration-dependent, since it is a recovery under the 2,048-token cap (all but one of the 99 affected responses are incomplete). For longer horizons, Sec. 3.3 names two gaps that compound with depth, and in the two-node extension the gated policy becomes almost inactive (1% reach, Table 11). We therefore expect learned routing to be harder at depth.

On the model configuration, we ran Qwen3-8B with thinking enabled as an exploratory check on 1,000 main-evaluation questions with a single replay. Thinking tokens are charged.

Qwen3-8B, 1,000 questionsThinking disabled (paper)Thinking enabled
Stop success62.8%66.0%
Output tokens, Stop / Coordination530 / 1,3633,097 / 8,077
Coordination minus Stop, success+2.7 points [0.8, 4.6]+4.7 points [3.2, 6.3]
Repair rate among initial errors16.4%16.8%
Harm rate among correct answers5.4%1.5%
Coordination minus Stop, utility+.0187 [−.0000, .0379]−.0028 [−.0180, .0131]

Thinking raises the success of Stop by 3.2 points [0.6, 5.8] at about six times the output tokens. Consultation still adds success under thinking, with the same repair rate and a much lower harm rate, but at 8,077 tokens per question its utility gain over Stop is −.0028 [−.0180, .0131]. In this run 11–13% of the initial and peer answers reach the 8,192-token cap and are counted as failures. The run suggests that the harm side of the break-even analysis depends on the reasoning configuration, and that the cost side then becomes the limiting factor. It is reported in App. A.8.

[Q3] Limitations of the learned routers or insufficient power

Thank you for this question, which the new experiments allow us to answer more precisely. The .110 penalty applies to the supplementary Hoeffding audit only. The conclusions about routers rest on paired intervals at n=2,000n = 2{,}000, whose standard error is about .0048 and which detect a gain of about .013 with 80% power, or .019 after correction over the twelve primary contrasts (Table R2). On the same questions they detect the correctness-bit policy (+.0508 [.0409, .0607]), the failure trigger (+.0079 and +.0059) and the saving of EMS3 over Vote (+.0032 and +.0036).

The held-out tests indicate that both explanations apply, to different policies. For confidence-based triggering, the main-sample intervals contain the gain later observed on held-out data. The gain of the break-even trigger is +.0087 [−.0031, .0206] and +.0053 [−.0066, .0169] on the main sample, where it is post hoc, and +.0163 [.0071, .0257] and +.0124 [.0026, .0220] over the 3,131 held-out questions, which is at the edge of what the main study could detect. For the selected learned router, the limitation appears to lie in the router. Its gain is +.0034 and +.0012 on the main sample and +.0064 [−.0003, .0133] and +.0029 [−.0043, .0100] on the held-out questions, so that it remains unestablished on a held-out sample one and a half times the size of the main one. Treating the 3,131-question pool as the target, design-based intervals on the main sample are about 40% narrower and lead to the same reading for the router ([−.00223, .00905] and [−.00427, .00665]; response to Reviewer G25b). We now state in Sec. 5.1 and in the conclusion what "inconclusive" covered and what the held-out tests resolved.

We hope that this clarifies what the study adds to the evidence that you cite. We would be glad to answer any further questions.

Thank you for clarifying the audit resolution and providing the new held-out tests.

However, the positive held-out result belongs to the frozen confidence classifier (from which the break-even trigger differs by only .0002–.0017), its advantage over always consulting remains unreported, and the headroom lower bound relies on an oracle that may implicitly use the reference answer (App. G). Because the points regarding sample size and selection bias are also general statistical properties, I still do not see a clear insight beyond the cited work on harmful communication and hard-to-exploit routing headroom.

Moreover, text-feature and embedding ridge routers achieve held-out gains comparable to the trigger, suggesting that the negative result for the selected router reflects selection across 982 configurations on 200 development questions rather than an inherent limitation of learned routing.

I therefore keep my score.

AuthorsOpenReview ↗

Response to Reviewer 6BrR’s follow-up comment

Thank you for the follow-up. We agree that the frozen confidence classifier accounts for most of the held-out gain. The break-even trigger uses the same score and sets its threshold on the development data. The empirical question was whether that threshold identifies questions on which consulting is worth its cost.

On the 2,000-question confirmation set, the split by whether the trigger fired was fixed in advance. Where it fired, consulting improved utility over Stop by +.0251 [.0044, .0457] on Qwen and +.0368 [.0093, .0640] on Gemma. Where it did not fire, the estimates were −.0026 [−.0139, .0095] and −.0059 [−.0188, .0070] (Table R6). Hu et al. (2026) report that high entropy among the three sampled votes indicates cases where debate is less likely to overturn a correct answer, but it does not identify cases where debate helps. Our study examines an earlier decision. Using only the initial answer and its confidence features, the frozen trigger selected a group in which consulting a peer yielded a positive average gain over Stop after accounting for token cost. In the least-confident group within the same set, consultation overturned 14.1% of initially correct Qwen answers and 24.1% of initially correct Gemma answers. Yet in a secondary analysis, consulting still had positive net value in that group on both backbones (+.0487 [.0123, .0848] and +.0468 [.0052, .0903]), after accounting for those harms and the added tokens.

Against always consulting, the trigger used 45–57% of the consultations across the two held-out sets, and observed success differed by at most 0.4 percentage points. Across all 3,131 questions, the estimated utility differences favored the trigger by +.0025 on Qwen and +.0046 on Gemma, and the classifier by +.0008 and +.0038. These are descriptive differences. The prespecified tests establish gains over Stop on Qwen in both tests and on Gemma in the confirmation test (Table R5), and they leave superiority over always consulting unresolved.

We also agree that the ridge results narrow the scope of the conclusion about learned routing. Four of the eight secondary, unadjusted held-out intervals for the ridge routers exclude zero. Reusing 200 development questions across 982 configurations is a plausible explanation for the selected router's weaker results, though our experiments do not isolate its effect. Finally, the cross-replay selector uses a reference-scored replay to choose an action. Its lower bound applies to an unrestricted oracle, and the held-out policy comparisons measure what the tested routers recover using information available at inference time. We will add the direct comparison with always consulting to App. A.7 and clarify the scope of the learned-router result in Sec. 5.1.

Reviewer_YGUPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks when an LLM agent that has already given an answer should consult a peer instead of returning that answer. The authors define the Causal Value of Coordination (CVC) as the expected difference in cost-adjusted utility between a coordination action and a reference action, starting from the same history and following a fixed continuation policy. With the incumbent policy as the reference, CVC is the standard advantage function. They estimate CVC by replaying both actions from shared histories, train ridge and MLP routers on text, embedding and confidence features, and evaluate the frozen routers on 2,000 MMLU-Pro questions with Qwen3-8B and Gemma-3-12B-it. Supplementary experiments on GSM8K and MMLU-Pro include held-out audits. The learned routers show no significant gain over always returning the initial answer, while a diagnostic that knows whether the initial answer is correct gains about .05 in utility. A rule for truncated or unparsed answers gives small gains, an existing early-stopping rule saves voting cost without changing answers, and no candidate passes the audits.

Critical strengths

The evaluation design is careful. Both actions are replayed from the same history, router selection is kept separate from the held-out evaluation, and frozen policies are compared end to end. The decomposition of coordination into repairs, harms and added cost (Figure 4) also shows clearly why predicting an initial error does not tell us whether consulting a peer helps. I appreciate that the negative results are reported openly.

Critical weaknesses

I do not see a clear scientific or methodological contribution. The proposed CVC is a difference between two cost-adjusted action values, and the authors acknowledge that it becomes the standard advantage function when the incumbent policy is used as the reference. The theoretical results follow existing policy-improvement and concentration bounds, while the paired replay procedure is closely related to C3 (Chen et al., 2026b). The main observation that predicting an incorrect answer does not necessarily tell us whether seeking help is beneficial is also well established in learning-to-defer research (e.g., Mozannar and Sontag, 2020), which the paper does not discuss. I find it difficult to identify what new problem or method this paper contributes beyond these existing ideas, and this is the main reason for my rating.

The experiments are too naive and do not provide convincing evidence for the proposed approach. The selected routers fail to establish improvements in the main experiments, and their utility point estimates are even lower than always coordinating on both backbones. Apart from a simplified version of EMS, the LLM experiments do not compare CVC. The only comparison with such methods is a toy scikit-learn proxy in Appendix H, which says little about LLM agents. Much of the experimental analysis focuses on statistical bounds and diagnostics, while the more basic questions of why the routers fail and which parts of the coordination setup actually matter remain unanswered. As a result, I do not think the experiments establish a meaningful finding about when LLM agents should coordinate, and this also weighs heavily in my assessment.

Questions for authors

The peer and the aggregator appear to use the same backbone as the initial agent. How would the results change with a different or stronger peer model?

Why should the features used here predict whether coordination will help? I could not find a clear hypothesis about which questions would benefit from coordination.

The paper discusses several existing learned routing and coordination methods, but the main experiments mostly compare against simple policies and variants of the proposed router. Why are these methods not compared on the LLM tasks under matched actions and inference budgets, beyond the scikit-learn experiments in Appendix H?

The confidence-based error classifier, which predicts initial correctness rather than coordination value, has higher utility gain point estimates than the selected CVC router on both backbones. What evidence supports predicting CVC instead of initial correctness as a more effective approach to routing?

The router is trained using only four stochastic replays per question. How consistent are the estimated CVC values across independent sets of replays for the same question? Without this information, it is difficult to tell whether the poor routing performance comes from noisy supervision or from the prediction method itself.

3 comments
AuthorsOpenReview ↗

Response to Reviewer YGUP (1/3)

We sincerely thank Reviewer YGUP for the detailed and candid review. The questions on the peer model, on the comparison with existing methods and on the consistency of the replay labels led us to run several additional experiments, which we report below. Tables R1, R5 and R6 are in the general response.

[W1] Contribution, and the relation to the advantage function, C3 and learning to defer

Thank you for these comments. We agree with the factual points. The paper states that incumbent-reference CVC is the standard advantage (Sec. 3.2), that the theory applies existing policy-improvement results (Sec. 3.3), and that the fixed-history replay of C3 is the closest procedure (Sec. 2). The paper is a measurement study, and its contribution lies in what the measurements show.

We are grateful for the pointer to learning to defer, which we had not discussed and have now added to Sec. 2. In that literature the optimal rule defers where the expected loss of the expert is below that of the model, which need not be where the model is uncertain (Mozannar and Sontag, 2020). Our setting differs in that the "expert" is an independent sample from the same model, merged with the initial answer by an LM aggregator at a cost that varies by question. Whether a region worth deferring on exists is then an empirical question, and our data answer it in two steps. Table R1 shows that the harm rate among correct answers rises from about 1% to 20–25% as confidence falls, so that the error rate needed to break even rises from about 12% to 61–62%. The prespecified held-out tests then locate such a region in the least confident answers. On the confirmation set, consulting gains +.0251 [.0044, .0457] and +.0368 [.0093, .0640] where a break-even trigger fires, and −.0026 [−.0139, .0095] and −.0059 [−.0188, .0070] where it does not (Table R6). Following your comment, we also fitted a cost-sensitive learning-to-defer router on our action set, and its results are reported under Q3.

[W2] The experiments, routers below always-coordinate, and why the routers fail

Thank you for these observations. The point estimates are as you describe and are reported in Table 2. On the main sample the selected routers are .0037 and .0007 below always-coordinate, which is well inside interval half-widths of about .009. We do not present the CVC router as a winning method. Over the two held-out sets together it gains +.0064 [−.0003, .0133] and +.0029 [−.0043, .0100] over Stop, whereas a break-even trigger built on the confidence classifier gains +.0163 [.0071, .0257] and +.0124 [.0026, .0220] (Table R5).

We believe that the experiments should be judged by whether they give a correct account of when consultation pays, and we have tried to strengthen that account in three respects. On why the routers fail, Table R1 gives a decomposition. On the main sample, repairs contribute about +.055 on each backbone, and harms and cost remove .047 and .053 of it, with the harm concentrated at low confidence. The score of the selected router barely sorts questions by error rate (33–37% across its score bins on Qwen), so it does not locate the region where consulting pays. On which parts of the setup matter, we added a cross-backbone peer, a stronger peer and a thinking configuration (Q1 below, and the response to Reviewer 6BrR). On the comparison with existing methods, we added two further decision rules to the LLM experiments (Q3 below).

AuthorsOpenReview ↗

Response to Reviewer YGUP (2/3)

[Q1] A different or stronger peer

Thank you for this question. The same-backbone setting was a deliberate scope, since we asked what a fixed set of agents can gain by consulting, but we agree that the submission left the question open. We ran three additional analyses. Gains are in percentage points of success over Stop with 95% intervals. We do not compare utility across backbones, because token counts depend on the tokenizer.

Initial answerPeerPolicyGain over Stop
Qwen3-8B2 Gemma answersVote and EMS3, cached responses−0.35 [−1.95, 1.20]
Gemma-3-12B-it2 Qwen answersVote and EMS3, cached responses+4.70 [3.10, 6.40]
Qwen3-8B2 Qwen answers (paper)Vote and EMS3+0.80 [−0.25, 1.85]
Gemma-3-12B-it2 Gemma answers (paper)Vote and EMS3+1.40 [0.35, 2.45]
Qwen3-8B1 Gemma answerAggregation, 2,000 new calls+2.50 [1.10, 3.90]
Gemma-3-12B-it1 Qwen answerAggregation, 2,000 new calls+3.95 [2.35, 5.60]
Qwen3-8B1 Qwen answer (paper)Aggregation+1.55 [0.20, 2.90]
Gemma-3-12B-it1 Gemma answer (paper)Aggregation+1.10 [−0.30, 2.50]
Qwen3-8BQwen3-32B answerAggregation, exploratory+7.3 [5.1, 9.5]
Gemma-3-12B-itQwen3-32B answerAggregation, exploratory+10.0 [7.5, 12.5]

Cross-backbone voting helps in the direction of the stronger model. A Gemma answer voted with two Qwen peers reaches 65.65%, which is about the accuracy of Qwen alone (65.45%), and a Qwen answer voted with two Gemma peers does not gain. EMS3 returns the same answers as Vote with 2.3 to 2.4 calls in place of 3. For aggregation, a peer from the other backbone exceeds a same-backbone peer by +0.95 [−0.50, 2.40] for a Qwen answer and by +2.85 [1.20, 4.60] for a Gemma answer. In the second case, the Qwen peer is also the stronger model, so the difference reflects both diversity and peer strength.

A stronger peer raises the repair rate from 16.4% to 28.0% (Qwen) and from 15.8% to 34.1% (Gemma), and the harm rate is 4.9% and 6.4%, against 5.4% and 8.9% with the same-backbone peer. However, the aggregate is no better than the stronger model answering alone (+0.9 [−1.1, 2.9] and +0.3 [−1.4, 2.1] points), and the aggregator returns the 32B peer's answer in 84–86% of questions. With a stronger peer, coordination therefore mostly defers to it, and the decision becomes one of model escalation, as studied by FrugalGPT and RouteLLM. The stronger-peer run is exploratory, on 1,000 main-evaluation questions with a single replay. All three analyses are in App. A.8 (Table R9).

[Q2] The hypothesis behind the features

Thank you for asking, since the submission did not state the hypothesis explicitly. Consultation changes the outcome through two events, a repair (the initial answer is wrong and the aggregator adopts a correct peer answer) and a harm (a correct initial answer is abandoned). The feature groups were chosen with these events in mind. Confidence features (token log-probabilities, entropy and answer-letter probabilities) indicate whether the answer is wrong and how contested it is. Question and answer embeddings capture topic and difficulty, as a proxy for whether an independent sample tends to be right. Response properties capture observable failures such as truncation.

The working hypothesis was that the value of consulting concentrates on low-confidence answers. The data support a narrower version of this hypothesis. Low confidence raises the error rate, but it also raises the harm rate and the cost (Table R1), so the break-even error rate rises with it, and the gain from consultation is concentrated in the least confident answers. The held-out tests show a gain in that region (Table R6). The learned value routers did not exploit this, although the confidence features were among their inputs. We now state the hypothesis in Sec. 4 and its outcome in Sec. 5.2.

AuthorsOpenReview ↗

Response to Reviewer YGUP (3/3)

[Q3] Comparison with existing routing and coordination methods on the LLM tasks

Thank you for this suggestion. The LLM experiments implement the decision rules of the closest families for a single action set, so that actions and budgets match by construction. This mapping was spelled out only for the proxy in App. H, and it is now given for the LLM experiments in Table R4 (App. D), with two rules added during the discussion period. Entries are gains over Stop for Qwen and Gemma. A dagger marks a pointwise 95% interval that excludes zero, without adjustment.

FamilyImplementationMain, 2,000Prospective, 1,131Confirmation, 2,000
Error-triggered debate, as in iMADConfidence classifier+.0090, +.0056+.0190†, +.0049+.0122†, +.0154†
Direct action-value routingSelected value router+.0034, +.0012+.0054, +.0048+.0069, +.0018
Best-sibling credit, as in TreeCreditNew: best-of-replay labels+.0089, +.0066+.0217†, +.0027+.0079, +.0139†
Learning to deferNew: cost-sensitive reduction+.0120†, +.0039+.0209†, +.0012+.0126†, +.0057

For EMS (Liu et al., 2026), EMS3 gains +.0032 and +.0036 over Vote with identical answers. The two new routers were fitted on the 850 training questions, selected on the 200 development questions, and frozen before scoring. They are post hoc on the main sample, where neither separates from the selected router: the paired differences are about +.005 on both backbones for best-of-replay and +.009 and +.003 for learning to defer, and all four intervals include zero. They were prespecified as secondary policies in the two held-out tests, where their results differ between the backbones.

[Q4] Predicting CVC or predicting initial correctness

Thank you for this question. We agree with its premise: our data do not show an advantage for CVC labels as a training target, and the new results support the simpler error-based trigger. On the main sample, the confidence classifier exceeds the selected router by +.0056 [−.0043, .0156] and +.0044 [−.0057, .0147]. On held-out data, the break-even trigger, which is this classifier with a different threshold, exceeds the router by +.0151 [.0019, .0283] (Qwen, prospective) and +.0147 [.0049, .0250] (Gemma, confirmation), with +.0070 [−.0018, .0159] and +.0002 [−.0124, .0134] in the other two cases. The estimated difference thus favors the error-based trigger in all four cases, and its interval excludes zero in two of them. Sec. 5.2 and App. A.7 now report this comparison.

We keep two distinctions. As an estimand, CVC is needed whichever target is used. The gain of the classifier over Stop is itself a CVC contrast, and paired replay is what shows that AUC .745 and .774 for initial error are worth about .012 to .016 in utility and where that gain is located. As a training label, the value contrast is the right target in principle (Cor. 1) but harder to learn. It is a small mean difference with a per-question standard deviation near .3 and a reliability of 0.74, whereas a correctness label is observed once without replay noise.

[Q5] Consistency of CVC estimates across replay sets

Thank you for this suggestion. We computed the consistency directly by splitting the four replays of each question into two halves. For the Coordination-minus-Stop utility, the training target of the router, the correlation between the halves is 0.59 on both backbones. This gives a reliability of 0.74 [0.68, 0.79] on Qwen and 0.74 [0.69, 0.78] on Gemma for the four-replay mean, and 0.41 for a single replay.

The supervision is therefore noisy, and about a quarter of the variance of the training label is replay noise. We do not think that this is the whole explanation, for three reasons. The post hoc cross-replay selector, which picks an action from one replay and is scored on another, gains +.0115 [.0035, .0214] and +.0312 [.0173, .0471] (Table 19), which suggests that differences in value persist across independent replays. More replays do not help: four-against-one gives −.0009 and −.0051, and eight help in 2 of 8 comparisons. Finally, the correctness-bit policy, trained on the same four-replay labels, gains +.0508 [.0409, .0607]. Together with the observation that the router's score does not sort questions by error rate (W2), this suggests that the prediction step limits the routers more than the labels do. The reliability analysis is reported in App. A.8.

We hope that these answers and the additional experiments address your concerns. We would be glad to answer any further questions.

Reviewer_nnHjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks and discusses a key research topic when an LLM agent that already has an answer should ask a peer for help. The authors define the Causal Value of Coordination (CVC), that is the expected cost-adjusted utility of coordinating, minus that of a reference action, under a fixed continuation policy. With the incumbent policy as the reference, this is the usual advantage, and the authors connect it to the performance-difference lemma. They estimate CVC with paired counterfactual replay, train routers on these estimates, and evaluate the frozen routers on held-out questions, including statistical audits.

The main experiments use Qwen3-8B and Gemma-3-12B-it on MMLU-Pro. The selected routers have small positive point estimates over always stopping, but the adjusted intervals include zero, and the supplementary audits do not certify positive gains for the learned candidates. The paper also reports repair/harm breakdowns, which is a privileged-information diagnostics, simple stopping and observable-failure controls, and sensitivity analyses of routing and evaluation.

Critical strengths
    1. Relevant problem and well-defined measurement target. The paper addresses an important question in adaptive multi-agent inference: when consultation provides enough benefit to justify its cost. By specifying the reference action, continuation policy, and cost-adjusted utility, CVC provides a clear target for studying coordination decisions. I find this formulation useful because it distinguishes predicting initial-answer errors from estimating whether consultation would actually help.
    1. Careful evaluation and informative diagnostics. The authors separate development selection from held-out policy evaluation and report statistical uncertainty rather than relying only on point estimates. The repair/harm analyses and evaluation-sensitivity results are particularly informative. For example, Table 19 illustrates how selecting a router using evaluation outcomes can produce a misleading impression of improvement. These analyses make the paper valuable as a measurement study, even without a successful learned router.
Critical weaknesses
    1. Unclear incremental contribution of the framework. For the incumbent-policy reference, CVC is an advantage, while paired replay, held-out evaluation, and concentration bounds build on established methods. This is not a problem by itself, but I am not yet convinced that their integration provides sufficiently consequential new insights or evaluation capabilities. The experiments do not establish a clear advantage for CVC-based supervision, and although the audits offer additional guarantees, their practical benefit remains unclear given the sample requirements discussed in Appendices C.4 and F. This uncertainty about the framework's incremental value is my main concern regarding the paper's significance.
    1. Limited distinctiveness and explanatory scope of empirical findings. The cost-adjusted repair/harm analyses are useful, but the underlying trade-off has been recognized in prior work such as iMAD (Fan et al., AAAI 2026). The routing diagnostics add useful evidence, including the replay-count and training-size ablations, but do not clearly identify what limits learned routing. As a result, I remain unsure whether the findings provide substantially new understanding of adaptive coordination or mainly reinforce existing observations and general evaluation cautions.
Questions for authors

I am uncertain about the significance of the contribution rather than its technical soundness. I would appreciate more clarification, particularly based on the existing results, that could help me better understand the framework's incremental value and the broader lessons of the study.

    1. Incremental value of the framework. Could the authors clarify what additional insights or evaluation capabilities the integrated CVC framework provides beyond confidence/error-based proxies and standard cost-sensitive held-out policy evaluation? A concrete example from the existing results would be particularly helpful. I would also be interested in how the authors interpret the point-estimate comparison in Section 5.2, considering the uncertainty and any differences in features, budgets, or selection procedures.
    1. Need more empirical insights. Could the authors elaborate on which empirical findings they consider most distinctive or informative, and what broader lessons might follow for adaptive coordination? I am also curious whether the existing diagnostics provide further insight into why the evaluated learned routers show limited gains, or whether the evidence remains inconclusive on this question.
    1. Practical value of statistical auditing. As a secondary clarification, I recognize that concentration-based audits can provide guarantees different from approximate paired confidence intervals, and that Appendices C.4 and F already examine the sample sizes needed to certify gains of the observed magnitude. Could the authors comment on when these additional guarantees would be particularly useful in practice, given the sample requirements?
4 comments
AuthorsOpenReview ↗

Response to Reviewer nnHj (1/2)

We sincerely thank Reviewer nnHj for the careful review, and for distinguishing the question of significance from that of technical soundness. Below we address each point with the results in the submission and with the experiments added during the discussion period. Tables R1, R2, R5 and R6 are in the general response.

[W1 & Q1] The incremental value of the framework beyond error proxies and standard held-out evaluation

Thank you for this question, which we regard as the central one for the paper. We agree that each ingredient is established, and Sec. 3.2 says so. Our answer is that the combination shows things that neither an error proxy nor a single held-out score can show. We give three concrete examples.

The first is the break-even analysis. An error proxy asks whether the initial answer is wrong, whereas paired replay also records what consultation does to answers that were right. Splitting Table 4 by initial correctness (Table R1) shows that the harm rate among correct answers rises from about 1% in the most confident bin to 20–25% in the least confident one, so that the error rate needed to break even rises from about 12% to 61–62%. This can only be seen with the consulted branch for every question, including the questions that were answered correctly. During the discussion period we tested the rule that follows from this analysis in two prespecified held-out tests (Table R5). A break-even trigger gains +.0139 [.0028, .0254] and +.0165 [.0044, .0291] over Stop on the confirmation set, and the gain is located where the rule fires (Table R6). The trigger is the confidence classifier of the paper with its threshold set by the break-even rule.

The second example concerns the reference. On GSM8K (Table 7), consultation is better than self-review by +.028 [.006, .058] and is indistinguishable from stopping (+.002 [−.003, .009]), while self-review is worse than stopping (−.027 [−.057, −.007]). The statement that consultation helps is therefore supported against one reference and not against another, and naming the reference and the continuation makes this explicit.

The third example is an evaluation capability. Freezing the candidates before evaluation changes the conclusion. In the supplementary evaluation (Table 19), a router chosen on evaluation outcomes shows +.0056 and +.0107 with intervals excluding zero, whereas the frozen development-selected candidate shows −.0004 and −.0001. Similarly, a 200-question pilot gave +.077 [.035, .122] for consultation on Qwen, against −.006 at evaluation (App. F).

Regarding the point estimates in Sec. 5.2, we did not read the larger estimates of the confidence classifier on the main sample (+.0090 and +.0056, against +.0034 and +.0012 for the selected router) as evidence for either training target. The paired differences are +.0056 [−.0043, .0156] and +.0044 [−.0057, .0147], and the two candidates also differ in their features (13 confidence features against 2,068 dimensions), model class and threshold grid. The held-out tests now give more information. Over the two held-out sets together, the confidence classifier gains +.0146 [.0057, .0239] and +.0116 [.0019, .0213] over Stop, while the selected router gains +.0064 [−.0003, .0133] and +.0029 [−.0043, .0100]. The break-even trigger exceeds the router by +.0151 [.0019, .0283] on Qwen in the prospective test and by +.0147 [.0049, .0250] on Gemma in the confirmation test. In the other two cases the differences are +.0070 [−.0018, .0159] and +.0002 [−.0124, .0134]. The estimated difference thus favors the simpler error-based trigger in all four cases, and its interval excludes zero in two of them. Sec. 5.2 and App. A.7 now report this comparison.

AuthorsOpenReview ↗

Response to Reviewer nnHj (2/2)

[W2 & Q2] The most distinctive findings, and what limits learned routing

Thank you for raising this point, and for the reference to iMAD. iMAD recognizes that debate can overturn correct answers and learns when to trigger it. Our data add where the overturned answers sit relative to the signal that such triggers use. They concentrate in the low-confidence region, on the main sample and on both held-out sets (Tables R1 and R6). This is why the break-even error rate is high there, and why the gain from consultation is concentrated in the least confident answers. We consider this finding, together with its prespecified test, the most distinctive one, and we now state it in Sec. 5.2.

On what limits learned routing, the diagnostics are more specific than the submission conveyed, and the new analyses add to them. First, there is evidence of item-level differences in value. The post hoc cross-replay selector, which chooses with one replay and is scored on an independent one, gains +.0115 [.0035, .0214] and +.0312 [.0173, .0471] on the supplementary evaluation (Table 19). Second, label noise is present, but it does not account for the result alone. The reliability of the four-replay value label is 0.74 on both backbones and 0.41 for a single replay (new, App. A.8), going from one to four replays changes utility by −.0009 and −.0051, and the correctness-bit policy trained on the same labels gains +.0508 [.0409, .0607]. Third, the score of the selected router barely sorts questions by their error rate. Across its own score bins the error rate is 33–37% on Qwen and 34–45% on Gemma, against 12–63% and 8–70% across the bins of the confidence classifier, although the confidence features are among the router's inputs. We also fitted two further routers on the cached replays, one with best-of-replay labels and one with a cost-sensitive learning-to-defer reduction, and neither separates from the selected router on the main sample (response to Reviewer YGUP, Q3). The evidence points to the prediction step more than to the labels.

As broader lessons, we would suggest the following. A trigger for consultation should be evaluated against the harm that it causes as well as the errors that it finds, since both increase as confidence falls. Claims of gains from adaptive coordination should name the reference and freeze the policy before evaluation. In our setting, the policies with established gains are a simple confidence-based trigger and structural rules such as EMS3 and failure recovery, and the learned value routers are not among them.

[Q3] When concentration-based audits are useful in practice

Thank you for this question. We agree that at 1,000 to 2,000 questions the bounds certify only large effects. The penalty is .110 for the supplementary Hoeffding audit and .019–.025 for empirical Bernstein on the four audited routers, and no candidate passes, also under the design-valid versions computed for Reviewer G25b. We see three situations in which the additional guarantees are useful.

The first is a change with a narrow range. EMS3 differs from Vote by a value in [0, .0205] by construction, and an exploratory range-aware bound supports its saving at our sample sizes (App. F). Answer-preserving optimizations such as early stopping, pruning and caching are of this kind. The second is deployment-scale logs. Certifying +.01 needs about 92,000 questions under Hoeffding (App. C.4), and for gains of about .012 empirical Bernstein at the observed variance needs 7,100–7,500 (App. F). This is large for a benchmark and small for production traffic, and the guarantee then holds at finite nn over a declared number of repeated updates, which a bootstrap interval does not provide. The third is a gate against harm. A harmful update of −.02 is accepted 0.185% of the time at n=180n = 180, but 52% of the time if replays are counted as independent observations (App. C.4).

For benchmark-scale claims the paired intervals are the informative instrument, and the held-out tests illustrate this. The gain of about .012 to .016 that they show is below what any of the bounds could certify at these sample sizes. Table R2 gives the gain that each instrument can detect, and we now say this in Sec. 5.4.

We hope that these clarifications and the new experiments address the concern about the incremental value of the study. We would be glad to answer any further questions.

Thank you for the detailed responses and additional experiments. I went through the rebuttal and the revised manuscript, and I found several of the clarifications helpful.

The break-even analysis is a useful addition. In particular, the conditional repair/harm breakdown shows that consultation harm increases with predicted initial-error risk, which helps explain why consultation gains can remain small even when the initial answer is likely to be wrong. The new held-out tests also provide stronger evidence that a confidence-based trigger can achieve positive gains in the evaluated setting. I also appreciate the clarification on audit resolution and the scope of the findings.

My remaining concern is about the significance of the incremental contribution. The break-even trigger uses the same confidence classifier as the existing baseline, and the direct gains from changing its threshold are small (0.0002–0.0017 in Table R5). The selected CVC-supervised router is also not established over Stop on any set. I recognize that improving learned routing is not the main goal of a measurement study, but these results suggest that the main contribution lies in the repair/harm analysis and its interpretation, rather than a new routing capability.

I find the conditional harm analysis useful, and I take the authors' point that it locates overturned correct answers relative to the trigger signal, which goes beyond what iMAD reports. Still, given Wynn et al. (2025) on debate overturning correct answers and Hu et al. (2026) on cases that are debate-safe but not debate-useful, I see this as a quantitative refinement of existing observations rather than substantially new understanding. The main conditional finding is also validated within one consultation setting (same-model peer, two backbones, one main benchmark), and the exploratory runs in App. A.8 suggest the harm–repair balance shifts considerably with thinking enabled or a stronger peer, so its broader implications remain somewhat uncertain.

Overall, the rebuttal has made the empirical contribution clearer, and I see this as a careful and transparent measurement study. However, relative to prior work on harmful consultation, the new knowledge still seems narrow to me, so I am keeping my rating at 2 for now, and will remain open to revisiting in the discussion.

Thank you so much for the follow-up. We agree with your reading of where the contribution lies. The paper is a measurement study and claims no new routing capability: the trigger is the existing confidence classifier, the change of threshold is worth .0002 to .0017, and the selected value router is not established on any set. The contribution is the repair/harm analysis and what it explains.

We would like to explain why we see that analysis as more than a refinement of Wynn et al. (2025) and Hu et al. (2026). What it adds is the reason an error signal converts poorly into routing value. As confidence falls, the repair rate shows no trend (14–19% on Qwen, 8–18% on Gemma) while the harm rate rises from about 1% to 20–25% (Table R1). Lower confidence, therefore, yields more errors to repair, but it does not make consulting better at repairing them, and it makes each correct answer more fragile. This is why a classifier with AUC .745 and .774 for initial error yields .012–.016 in utility where knowing correctness would yield about .05, and why the error rate needed to break even rises from 12% to 61–62%. The same accounting gives a rule that can be tested, and the prespecified split of the confirmation test supports it: a signal available before any further sample is drawn identifies a group of questions on which consulting pays (Table R6), which is the debate-useful side that Hu et al. report vote entropy does not find.

Regarding scope, we agree that the rates are setting-specific, and Table R3 lists which findings depend on the configuration. What we expect to transfer is the accounting, and the exploratory runs in App. A.8 show how it reads a new setting. With thinking enabled, the harm rate falls from 5.4% to 1.5%, which lowers the break-even error rate, but the added cost rises from about 830 to about 4,980 output tokens, which raises it more. Computed from these rates, it moves from about 29% to about 35%, against error rates of 37.2% and 34.0%, and the gain of consulting goes from +.0187 [−.0000, .0379] to −.0028 [−.0180, .0131]. The limiting factor changes from harm to cost. With a stronger peer, the repair rate rises from 16% to 28–34% at a harm rate of 5–6%, so consulting broadly increases success, and the decision becomes one of model escalation. The balance shifts in both runs, as you note, and the three measured rates say why. We will add this calculation to App. A.8.

We would be glad to provide any further analysis that would help.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers (1/3)

We sincerely thank all four reviewers for the careful and constructive reviews. We appreciate the encouraging comments on the formulation of the target, on keeping selection separate from held-out evaluation, and on the reporting of negative results. A question that appears in several reviews is what the study teaches beyond its known ingredients. We would like to address it first. During the discussion period we carried out two held-out tests on 3,131 questions outside the main sample, which bear directly on this question, together with a series of additional analyses suggested by the reviews. All changes are highlighted in blue in the revised manuscript.

We agree that the ingredients are established. For an incumbent reference, CVC is the standard advantage (Sec. 3.2), and the replay, the evaluation and the bounds build on existing tools. The paper is a measurement study, and we believe its contribution should be judged by what the measurements show. Below we describe one analysis that the submission did not draw out, and two prespecified tests of it.

A break-even analysis of consultation (Table R1). Paired replay records both branches for every question, so the confidence bins of Table 4 can be split by initial correctness. For a bin with error rate ee, repair rate rr, harm rate hh and added cost cc, the gain of consulting is e⋅r−(1−e)⋅h−ce \cdot r - (1-e) \cdot h - c, which reproduces every ΔJ\Delta J in Table 4, and consulting pays only where ee exceeds (h+c)/(r+h)(h+c)/(r+h). Entries are Qwen3-8B / Gemma-3-12B-it on the 2,000 main evaluation questions. Bins are the development-fixed quintiles of predicted initial error.

BinError rateRepair rate among errorsHarm rate among correct answersBreak-even error rateGain of consulting
1 (lowest)12.2% / 8.2%16.7% / 11.1%1.4% / 1.0%11.7% / 12.2%+.0008 / −.0049
220.6% / 20.5%14.5% / 16.7%4.1% / 3.1%25.5% / 19.7%−.0092 / +.0015
328.7% / 38.1%16.3% / 8.2%3.5% / 6.9%21.2% / 50.8%+.0149 / −.0191
442.1% / 54.4%19.2% / 12.8%7.5% / 14.6%31.1% / 56.9%+.0292 / −.0066
5 (highest)62.8% / 70.4%14.2% / 17.8%19.6% / 24.8%61.4% / 61.7%+.0045 / +.0373

Over all questions, consulting repairs 15.8% and 14.1% of initial errors and overturns 6.0% and 7.2% of correct answers. The harm rate rises from about 1% in the most confident bin to 20–25% in the least confident one, so the error rate needed to break even rises from about 12% to 61–62%. This explains why the gains from consulting are small. On this sample no bin has a gain whose 95% interval excludes zero (for bin 5, +.0045 [−.0304, .0406] and +.0373 [−.0045, .0798]).

Two prespecified tests of this analysis (new Table R5). The analysis suggests a rule: consult only where the predicted error rate exceeds the break-even rate. We implemented it as a break-even trigger, which is the frozen confidence classifier of the paper with its threshold set by this rule on the 200 development questions. Before any scoring, we prespecified a test of the trigger against Stop on the 1,131 pool questions that were not in the main sample (prospective test), and afterwards on 2,000 further MMLU-Pro questions (confirmation test). Every policy and threshold was frozen, nothing was refitted, and each backbone was scored once. As for the main evaluation, the protocol was fixed and recorded before the runs (App. A.7). Entries are gains over Stop with 95% intervals.

PolicyQuestionsQwen3-8BGemma-3-12B-it
Break-even trigger (primary)Prospective, 1,131+.0205 [.0048, .0366], Holm p=.022p = .022+.0050 [−.0106, .0209], p=.534p = .534
Break-even trigger (primary)Confirmation, 2,000+.0139 [.0028, .0254], Holm p=.017p = .017+.0165 [.0044, .0291], Holm p=.015p = .015
Break-even triggerBoth sets, 3,131+.0163 [.0071, .0257]+.0124 [.0026, .0220]
Confidence classifierBoth sets, 3,131+.0146 [.0057, .0239]+.0116 [.0019, .0213]
Always coordinateBoth sets, 3,131+.0138 [.0038, .0240]+.0078 [−.0036, .0192]
Selected routerBoth sets, 3,131+.0064 [−.0003, .0133]+.0029 [−.0043, .0100]

The rows for both sets together are pooled estimates rather than additional tests. The interpretation of these results follows in the next comment.

AuthorsOpenReview ↗

General response to all reviewers (2/3)

What the tests show. The break-even trigger gains over Stop on Qwen in both tests and on Gemma in the confirmation test. On the prospective test the Gemma estimate is smaller, +.0050, with an interval that includes zero. Over the two sets together the gain is about .012 to .016 on both backbones, which is close to the smallest gain that the main study could detect with 80% power (about .013, Table R2). On the main sample the same trigger, which is post hoc there, gives +.0087 [−.0031, .0206] and +.0053 [−.0066, .0169]. These intervals contain the held-out estimates, so the main-sample result for this contrast is consistent with limited resolution. The selected learned router does not separate from Stop on any set.

Where the gain is located (new Table R6). On the confirmation set the gain sits where the break-even rule places it.

Confirmation setQwen3-8BGemma-3-12B-it
Questions on which the trigger fires55.2%45.0%
Gain of consulting where it fires+.0251 [.0044, .0457]+.0368 [.0093, .0640]
Gain of consulting where it does not fire−.0026 [−.0139, .0095]−.0059 [−.0188, .0070]
Repair rate / harm rate where it fires16.2% / 8.6%19.3% / 15.6%
Repair rate / harm rate where it does not fire10.7% / 1.6%11.9% / 3.1%

On the prospective set the same holds for Qwen (+.0359 [.0081, .0637] where the trigger fires and −.0108 [−.0254, .0038] where it does not), while for Gemma the gain where it fires is +.0108 [−.0216, .0443]. The rise of the harm rate with predicted error repeats on both held-out sets, reaching 14–24% in the least confident bin, and that bin is where consulting pays: +.0487 [.0123, .0848] and +.0468 [.0052, .0903] on the confirmation set.

How this refines our conclusions. On the main sample, no prespecified primary contrast separates from Stop, and this is unchanged. The break-even analysis explains why the gains from consulting are small. A confidence-based trigger nevertheless has a positive value of about .012 to .016 on held-out data, and this value is concentrated in the least confident answers. The selected learned router is not established over Stop on any set. The break-even trigger exceeds it by +.0151 [.0019, .0283] on Qwen in the prospective test and by +.0147 [.0049, .0250] on Gemma in the confirmation test, and by +.0070 [−.0018, .0159] and +.0002 [−.0124, .0134] in the other two cases. We have revised the abstract, Sec. 5.2 and the conclusion accordingly, and App. A.7 reports both tests in full.

AuthorsOpenReview ↗

General response to all reviewers (3/3)

Two further findings from the submission. First, the paired replays bracket the headroom. A perfect error detector is worth +.0508 [.0409, .0607] and +.0504 [.0406, .0607] (correctness-bit policy, Sec. 5.2). In the supplementary evaluation, a post hoc selector that chooses between Stop and Coordination with one replay and is scored on an independent one gains +.0115 [.0035, .0214] and +.0312 [.0173, .0471]. In expectation this is a lower bound on the gain of the best history-measurable policy (Prop. 3). A five-action oracle on realized outcomes shows +.079 and +.083, an upper bound that also selects on replay noise (Table 19). Second, freezing candidates before evaluation changes conclusions. In the same supplementary evaluation, a router chosen on evaluation outcomes shows +.0056 and +.0107 with intervals excluding zero, against −.0004 and −.0001 for the frozen development-selected candidate.

Resolution of each instrument (Table R2). Several reviews ask what gain the evaluation can detect.

InstrumentnnSmallest gain it detects
Paired bootstrap interval, pointwise 95%2,000about .013 (80% power)
Paired bootstrap interval, corrected over the 12 primary contrasts2,000about .019 (80% power)
Empirical-Bernstein lower bound, main evaluation, four audited routers2,000penalty .019–.025
Hoeffding lower bound, main evaluation2,000penalty .062 (i.i.d. formula), .037 (design-matched)
Hoeffding audit, supplementary, K=3K = 31,000penalty .110

The gain observed in the held-out tests is therefore at the limit of what the paired intervals could detect on the main sample, and below what any of the bounds could certify.

Scope of each finding (Table R3).

FindingWhere it is observedDependence
The harm rate of consulting rises with predicted errorMain sample and both held-out sets, 2 backbonesHeld in every set; with thinking enabled the overall harm rate falls to 1.5% (exploratory)
A confidence-based trigger gains over StopQwen in both held-out tests, Gemma in the confirmation test, and both backbones over the two sets togetherAbout .012–.016, located in the least confident answers
The selected learned router is not established over its referenceMain sample and both held-out sets; four-action, post-peer and two-node decisionsHeld in every setting tested
Early stopping keeps Vote's answers and saves costMain evaluation (2 backbones) and supplementary evaluation (3 backbones)Structural for any majority-of-three vote
Failure-trigger gainMain sample; held-out sets together (+.0106 [.0071, .0143] and +.0068 [.0040, .0099])Recovery of responses cut at the 2,048-token cap
Value of always coordinating against StopQwen −.0059, +.0072 and +.0138 [.0038, .0240] (supplementary, main, held-out); Gemma +.0081, +.0019 and +.0078 [−.0036, .0192]Changes with backbone and sample
Voting and aggregation help the weaker answererMistral-Nemo (exploratory); a Gemma answer with Qwen peers; a Qwen3-32B peer (exploratory)Depends on the relative strength of the peer

Changes in the revision. New tables keep the labels R1 to R9 in the revised PDF, so the table numbers of the submission are unchanged.

  1. Abstract, Sec. 1, Sec. 5.2 and Sec. 6: the break-even analysis (Table R1) and the outcome of the held-out tests.
  2. App. A.7: the two prespecified tests (Tables R5 and R6), with the full protocol and the robustness checks.
  3. Sec. 5.1, Sec. 5.4 and App. A.5: the gain each instrument can detect (Table R2), design-based intervals (Table R7), and bounds valid under the sampling design.
  4. App. A.4: the measured overhead (Table R8).
  5. App. A.8: cross-backbone and stronger peers (Table R9), thinking mode, and label reliability.
  6. App. D: Table R4 lists every policy with its decision rule, its reference and the prior method it implements, including best-of-replay and learning-to-defer routers.
  7. Sec. 2: learning to defer (Mozannar and Sontag, 2020), Choi et al. (2026), Wynn et al. (2025), Kim et al. (2026) and Hu et al. (2026).
  8. App. A.6: Table R3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.