- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper defines the policy-relative Causal Value of Coordination (CVC). CVC compares the cost-adjusted utility of a coordination action with that of a feasible alternative. The comparison uses the same history and a fixed continuation policy. The authors use paired counterfactual replay to estimate differences in utility and cost. They train routers using text, confidence, and frozen embeddings. They then assess gains through full-policy evaluation on held-out data and audits based on lower confidence bounds. The main experiments evaluate Qwen3-8B and Gemma-3-12B-it on the same 2,000 MMLU-Pro questions. Additional experiments include GSM8K. The results show that confidence can predict whether the initial answer is incorrect.
Critical strengths
- The paper clearly distinguishes initial error prediction from assessing the value of coordination.
- The measurement and evaluation procedures are well structured, with attention to selection bias and the unit of analysis.
- The negative empirical findings are informative and include a detailed breakdown of outcomes.
Critical weaknesses
- The audit guarantees rely on an independent and identically distributed (i.i.d.) assumption. However, the main evaluation uses category-stratified sampling without replacement.
- Most evidence comes from question-answering tasks with fixed prompts and aggregation rules. The decision is whether to consult a peer after the initial answer. The appendix expands the action set and includes two-node routing. However, it does not provide systematic validation with heterogeneous agents, complex tool-use tasks, or long-horizon, multi-round coordination.
- The main cost metric does not capture the full online routing overhead.
Questions for authors
- How should confidence-bound coverage and detectable gains be interpreted under the current sampling design? Please explain how the i.i.d. approximation affects confidence-bound coverage under stratified sampling without replacement. Is there any validation or derivation that matches the actual sampling design? Given the current sample size and error budget, how large must a utility gain be for the audit to detect it reliably? This information will affect my interpretation of the audit guarantees and the fact that no candidate passed.
- Which conclusions generalize beyond the current coordination protocol? The paper already includes an expanded action set and two-node experiments. Please clarify which findings mainly depend on the current peer-generation process, aggregation rules, and task settings. Which findings are supported across settings? Are results available for heterogeneous peers or other coordination protocols under the same evaluation procedure? This will affect my assessment of the generalizability of the empirical findings and the scope of the paper’s contribution.
- Do the main policy comparisons still hold after including feature extraction and scoring costs? Please explain how including these additional costs changes each router’s utility relative to immediate stopping (Stop) in the main evaluation. The costs should include frozen embedding extraction, teacher-forced scoring, and input tokens. Are timing or cost measurements available for the full online execution path? These results will affect my assessment of cost-adjusted utility and the practical implications for deployment.
3 comments
Response to Reviewer G25b (2/3)
To check the sampling argument independently of the data, we also ran a simulation that matches the design. We drew 2,000 questions without replacement under the category quotas 20,000 times from a synthetic pool of 3,131 questions across 14 categories. The pool is calibrated to the selected router (coordination on 45.4% of the questions, per-question standard deviation of .21), and the spread of repair and harm rates across categories is assumed. No bound exceeded the pool mean in any draw. The reported 95% interval covered the pool mean in 99.9% of draws, and a design-based interval covered it in 94.7–95.3% of draws.
On the detectable gain, the reported intervals have a standard error of about .0048 and reach 80% power near a gain of .013, or .019 after correction over the twelve primary contrasts. For the four audited routers, the penalties are .019–.025 for empirical Bernstein and .037 for the design-matched Hoeffding bound, and the penalty of the supplementary audit is .110 (, ). The statement that no candidate passed should therefore be read as "not certified at this resolution", which is how Sec. 4 words a declined audit. The held-out tests added during the discussion period (Table R5) give a direct illustration. A confidence-based trigger gains about .012 to .016 over Stop on 3,131 further questions, which is at the limit of what the paired intervals could detect on the main sample and below what any of the bounds could certify.
[W2 & Q2] What generalizes beyond the current coordination protocol
Thank you for this question. Table R3 lists each finding with the settings in which it was observed, and we summarize it here.
Three findings are supported across the settings that we tested. The harm rate of consulting rises with predicted error on the main sample and on both held-out sets. EMS3 keeps the answers of Vote while saving cost, which is structural and holds on three backbones. The learned value router is not established over its reference, for the Stop, Vote and Review-then-Stop references and for binary, four-action, post-peer and two-node decisions.
Other findings depend on the peer, the aggregation rule, or the task. These are the value of always coordinating against Stop, the failure-trigger gain (a recovery under the 2,048-token cap), and the sensitivity to the scoring rule (on Qwen, +0.2 points under normalized scoring against +26.6 under strict extraction). For heterogeneous peers, we added three analyses during the discussion period (response to Reviewer YGUP, Q1, and Table R9). Because both backbones answered the same 2,000 questions, cross-backbone voting can be scored from cached responses. A Gemma answer voted with two Qwen peers gains +4.70 points [3.10, 6.40], while a Qwen answer voted with two Gemma peers gains −0.35 [−1.95, 1.20]. With 2,000 new aggregator calls per direction, aggregation with a peer from the other backbone gains +2.50 [1.10, 3.90] for a Qwen answer and +3.95 [2.35, 5.60] for a Gemma answer, against +1.55 [0.20, 2.90] and +1.10 [−0.30, 2.50] with a same-backbone peer. In an exploratory run with a Qwen3-32B peer, the repair rate roughly doubles, but the aggregate is not better than the stronger model answering alone. The benefit of a heterogeneous peer therefore follows its relative strength.
Other protocols under the same procedure are already in the paper, namely self-review, voting, early stopping, aggregation after a peer, a four-action menu, and two-node review-then-coordinate on GSM8K.
Response to Reviewer G25b (3/3)
[W3 & Q3] Policy comparisons once feature extraction and scoring costs are included
Thank you for this question. The primary objective counts output tokens only (App. A.4). We have now measured the overhead of the full online path on one H100 GPU with vLLM 0.11.0, using 200 questions for the scoring pass and the encoder, and 64 questions for generation.
| Measured per question | Qwen3-8B | Gemma-3-12B-it |
|---|---|---|
| Initial answer: prompt tokens / output tokens | 262 / 535 | 257 / 609 |
| Teacher-forced scoring pass: tokens / seconds | 797 / 0.058 | 866 / 0.104 |
| Encoder pass (Qwen3-Embedding-0.6B): tokens / seconds | 722 / 0.035 | 790 / 0.038 |
| Generating one initial answer, single stream: seconds | 5.59 | 11.24 |
The overhead is paid on every question, before the decision, and only by policies that compute features. Stop, Coordination, Vote, EMS3, the text-feature router (CPU features only), and the failure trigger pay nothing. The confidence classifier and the break-even trigger need the scoring pass, and the selected router needs the scoring pass and the encoder. The table below gives the gain over Stop on the main 2,000 questions under three charges.
| Policy | Output tokens only (paper) | Overhead charged by measured time | Overhead charged as output tokens |
|---|---|---|---|
| Qwen, selected router | +.0034 [−.0060, .0128] | +.0033 [−.0061, .0127] | −.0118 [−.0212, −.0025] |
| Gemma, selected router | +.0012 [−.0079, .0104] | +.0011 [−.0080, .0103] | −.0154 [−.0245, −.0061] |
| Qwen, confidence classifier | +.0090 [−.0026, .0207] | +.0089 [−.0026, .0206] | +.0010 [−.0106, .0127] |
| Gemma, confidence classifier | +.0056 [−.0063, .0173] | +.0056 [−.0064, .0172] | −.0030 [−.0149, .0086] |
In wall time, the overhead is small. The scoring pass and the encoder together take about 0.09 s on Qwen and 0.14 s on Gemma, against 5.6 s and 11.2 s to generate one initial answer, which corresponds to about 8 output tokens. Charged by measured time, every gain changes by .0001 or less. In token count, the overhead is not small, since the scoring pass alone re-reads more tokens than the initial answer generates. If each of these tokens were priced like a generated token, the selected router would fall below Stop on both backbones. The two charges bracket the cost: a forward pass over existing tokens is cheaper than generating them, but it is not free.
The same accounting applies to the held-out tests. Charged by measured time, the gain of the break-even trigger is unchanged. Under the upper-bound charge it becomes +.0125 [−.0030, .0287] and −.0036 [−.0191, .0122] on the prospective set and +.0059 [−.0052, .0173] and +.0078 [−.0044, .0204] on the confirmation set. This charge is deliberately pessimistic, since the scoring pass takes 0.058 s and 0.104 s per question, against 5.6 s and 11.2 s for generating one initial answer. On Qwen in the prospective test, the gain remains positive as long as a scoring-pass token costs less than 2.6 output tokens. We report these results in App. A.4 (Table R8) and with the held-out tests in App. A.7.
We hope this resolves questions about the audit guarantees, the scope of the findings, and cost accounting. We would be glad to answer any further questions.
Response to Reviewer G25b (1/3)
We sincerely thank Reviewer G25b for the precise and constructive questions. Each of them led to a concrete analysis and to a change in the paper. Tables R2, R3, R5, and R6 are in the general response.
[W1 & Q1] The i.i.d. assumption under category-stratified sampling without replacement
Thank you for pointing this out. The submission left the matter open, and we have now worked it out for the actual design, in which 2,000 questions are drawn without replacement from the remaining pool of 3,131 with category-proportional quotas, and the target is the pool mean.
The Hoeffding bounds remain valid, up to a quota-rounding term. Hoeffding (1963, Thm. 4) shows that, for any continuous convex f, the expectation of f of a sum sampled without replacement is at most its expectation under sampling with replacement. The bounds on the moment-generating function therefore carry over within each category. Since the categories are sampled independently, these bounds multiply, and the i.i.d. Hoeffding inequality holds for the stratified sample mean around its expectation. With integer quotas, this expectation differs from the pool mean by at most 0.7% of the largest gap between a category mean and the overall mean. The post-stratified estimate removes this term: the same argument applies to it with ∑hWh2/nh in place of 1/n, where Wh is the pool share of category h, and no point estimate changes by more than .00002. The same argument covers Prop. 2 whenever audit questions are drawn without replacement from a finite pool. Applying Serfling's finite-population refinement within each category (Bardenet and Maillard, 2015, Prop. 2.2) reduces the penalty from .062 to about .037, because 64% of the pool is sampled.
The empirical-Bernstein bounds were an approximation, since the Maurer–Pontil bound is proved for i.i.d. samples. We computed its analog for simple random sampling without replacement (Bardenet and Maillard, 2015, Thm. 4.3) from the per-question differences. For the four audited routers, its penalties are .030–.033, against .019–.025 for the bound in the paper, because its range term has a larger constant. It does not cover stratification, so we report it for reference, treat the Hoeffding bound as the design-valid one, and label the Maurer–Pontil values as approximate.
We also computed design-based intervals from the per-question differences, with category weights from the pool and the stratified finite-population standard error.
The design-based intervals are about 40% narrower than the bootstrap intervals, which treat the sample as i.i.d. and are therefore conservative for the pool mean. Under the design-based reading, a few contrasts exclude zero, including the confidence classifier against Stop on Qwen. These intervals condition on the realized responses and are not adjusted for multiplicity, so we report them as a sensitivity analysis and do not base conclusions on them. Every lower bound stays negative under every version, and no audit decision depends on the approximation. Table R7 in App. A.5 gives all primary and secondary contrasts.