- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LMCircuit treates LLM classifiers as imperfect tool for predicting labels and tries to combine their answers intelligently. It uses dynamic programming to decide which models to query and when to balance accuracy against cost. The authors provide theoretical results on how accuracy improves with additional information and show that LMCircuit outperforms several existing routing, cascading and voting methods across four benchmarks while remaining fast to train.
Critical strengths
- LMCircuit combines model selection, query ordering and stopping into one pricipled framework.
- It trains quickly and consistenly outperforms ten baselines across four diverse tasks.
- Paper explains why the method works, showing that is prioritizes models by information gained per unit cost and spends more queries when predictions are uncertain.
- The pool construction study offers practical guidance for selecting useful models.
Critical weaknesses
- Scaling-law analysis assumes instruments perform better than random guessing on every prompt but a substantial fraction of prompts in some benchmarks violate this condition. The theory’s applicability to these cases remains unclear.
- LMCircuit includes a supervised text-based classifier that can predict labels without asking any LLM but the paper does not report its standalone performance or compare against strong supervised stacking baselines under matched conditions.
Questions for authors
- Scaling-law analysis assumes that every instrument performs at least as well as random guessing on every prompt. How often is this assumption violated in the benchmarks and how robust are the theoretical predictions when such violations occur?
- How much of LMCircuit's improvement comes from its zero-cost classifier versus adaptive querying? Compare its standalone performance with the a supervised stacker under matched conditions to help isolate the contribution of adaptive querying.
3 comments
Thank you for additional experiments. The ablations address my concerns about adaptive querying and the zero-cost prior. I would appreciate (a) similar results for LMArena and HealthBench, (b) test accuracy at each operating point and (c) details on CI computation.
Regarding Q1, I agree with the flipping argument when accuracies are known. My remaining concern is robustness to misestimated per-prompt accuracies, particularly on systematically incorrect prompts. A brief mention in the limitations would suffice.
Thank you for the follow-up! We provide the additional experiments and statistical details below and will include them in Section C.4.4.
Q1. Yes, these theoretical results assume known instrument accuracies and do not address errors in estimating prompt-specific reliability. We will explicitly acknowledge this limitation in the paper.
Q2.
(a–b) Additional comparisons and test accuracy. The tables below report 95% confidence intervals for both test accuracy and relative cost savings.
BeaverTails
| Target (val) | LMCircuit cost | LMCircuit test acc | Stacker cost | Stacker test acc | Savings vs. stacker | Non-adaptive cost | Non-adaptive test acc | Savings vs. non-adaptive |
|---|---|---|---|---|---|---|---|---|
| 79% | 0.006 | 80.70% [79.27, 82.08] | 0.021 | 80.66% [79.28, 81.95] | 71% [64, 78] | 0.035 | 80.72% [79.34, 82.07] | 83% [80, 86] |
| 80% | 0.021 | 81.39% [79.97, 82.77] | 0.041 | 81.02% [79.64, 82.31] | 49% [43, 55] | 0.063 | 81.33% [79.96, 82.72] | 67% [60, 71] |
| 81% | 0.049 | 82.05% [80.73, 83.42] | 0.061 | 81.39% [79.95, 82.75] | 20% [3, 38] | 0.131 | 81.94% [80.64, 83.28] | 63% [49, 72] |
| 82% | 0.105 | 82.73% [81.43, 84.01] | 0.184 | 82.00% [80.66, 83.30] | 43% [25, 60] | 0.221 | 82.57% [81.28, 83.86] | 52% [37, 66] |
PolitiCAUSE
| Target (val) | LMCircuit cost | LMCircuit test acc | Stacker cost | Stacker test acc | Savings vs. stacker | Non-adaptive cost | Non-adaptive test acc | Savings vs. non-adaptive |
|---|---|---|---|---|---|---|---|---|
| 78% | 0.007 | 77.60% [76.13, 79.09] | 0.003 | 76.48% [74.90, 78.02] | −136% [−193, −63] | 0.037 | 77.71% [76.35, 79.15] | 80% [75, 86] |
| 79% | 0.024 | 78.09% [76.59, 79.60] | 0.046 | 77.66% [76.27, 79.13] | 49% [32, 65] | 0.059 | 78.42% [76.97, 79.90] | 60% [47, 73] |
| 80% | 0.079 | 78.66% [77.19, 80.10] | 0.155 | 78.53% [77.11, 80.02] | 49% [36, 62] | 0.193 | 78.98% [77.60, 80.44] | 59% [49, 68] |
| 81% | 0.435 | 80.11% [78.71, 81.48] | 1.391 | 79.89% [78.42, 81.36] | 69% [61, 76] | 0.771 | 80.16% [78.75, 81.51] | 44% [26, 57] |
LMArena
| Target (val) | LMCircuit cost | LMCircuit test acc | Stacker cost | Stacker test acc | Savings vs. stacker | Non-adaptive cost | Non-adaptive test acc | Savings vs. non-adaptive |
|---|---|---|---|---|---|---|---|---|
| 65% | 0.004 | 64.46% [63.46, 65.53] | 0.071 | 64.78% [63.87, 65.71] | 95% [90, 99] | 0.033 | 64.73% [63.75, 65.78] | 89% [86, 94] |
| 66% | 0.013 | 64.83% [63.83, 65.92] | 0.137 | 65.71% [64.81, 66.63] | 90% [86, 93] | 0.095 | 65.52% [64.61, 66.52] | 86% [81, 90] |
| 67% | 0.070 | 66.15% [65.19, 67.16] | 0.282 | 66.24% [65.28, 67.18] | 75% [71, 79] | 0.157 | 66.32% [65.35, 67.32] | 56% [49, 62] |
| 67.5% | 0.130 | 66.69% [65.74, 67.67] | 0.729 | 66.70% [65.71, 67.67] | 82% [79, 86] | 0.245 | 66.66% [65.71, 67.66] | 47% [33, 58] |
HealthBench
| Target (val) | LMCircuit cost | LMCircuit test acc | Stacker cost | Stacker test acc | Savings vs. stacker | Non-adaptive cost | Non-adaptive test acc | Savings vs. non-adaptive |
|---|---|---|---|---|---|---|---|---|
| 79.5% | 0.003 | 79.93% [78.81, 80.96] | 0.124 | 79.86% [78.83, 80.88] | 98% [97, 99] | 0.045 | 79.88% [78.78, 80.88] | 94% [93, 96] |
| 80% | 0.008 | 80.26% [79.16, 81.26] | 0.324 | 80.12% [79.08, 81.10] | 97% [97, 98] | 0.101 | 80.10% [79.03, 81.09] | 92% [90, 93] |
For completeness, we also report the highest test-accuracy achievable by each method, with their cost.
Highest-accuracy operating points
| Dataset | LMCircuit test acc / cost | Stacker test acc / cost | Non-adaptive test acc / cost |
|---|---|---|---|
| BeaverTails | 83.46% / 0.77 | 83.19% / 2.56 | 83.61% / 2.37 |
| PolitiCAUSE | 80.44% / 1.37 | 79.89% / 2.44 | 80.50% / 2.19 |
| LMArena | 67.10% / 0.29 | 66.76% / 1.69 | 67.00% / 0.81 |
| HealthBench | 81.05% / 0.67 | 80.82% / 1.49 | 80.88% / 2.33 |
Discussion. LMCircuit achieves lower query costs than the non-adaptive circuit at every reported target. Since both methods use the same learned belief model, these comparisons directly measure the benefit of adaptive classifier selection and stopping. The savings over supervised stacking are also substantial, except at the lowest PolitiCAUSE target, where the stacker is cheaper but has lower test accuracy.
(c) Confidence intervals. We compute 95% confidence intervals using 2,000 bootstrap resamples of the test set. For each resample, we compute test accuracy and relative cost savings, defined as , and report the mean cost saving. The reported 95% intervals are the 2.5th and 97.5th percentiles of these estimates. For each prompt, we average cost and correctness over eight independent draws of classifier responses. We use the same resampled test prompts for LMCircuit and each baseline. We also retrain the belief model used by LMCircuit and the non-adaptive circuit five times, and report the average performance.
Thank you for your thoughtful and detailed review! We are glad that you appreciated LMCircuit's principled framework, its consistent improvements across various baselines, and the theoretical insights explaining its performance. Following your suggestions, we added the new results to the paper in Section C.4.4 of the appendix. We welcome more questions or suggestions.
Q1: Scaling-law analysis assumes that every instrument performs at least as well as random guessing on every prompt. How often is this assumption violated in the benchmarks and how robust are the theoretical predictions when such violations occur?
For the theoretical analysis, the assumption ai(z)≥1/2 is without any loss of generality since an instrument with accuracy ai(z)<1/2 provides the same information, denoted by Ii(z) in Line 412, as one with accuracy 1−ai(z) after reversing its answers. Thus, the results also apply to instruments that perform worse than random guessing, provided their accuracies are known. The purpose of these results is to characterize the fundamental limits of the accuracy–cost tradeoff under known instrument accuracies, establishing what is achievable with optimal use of the available instruments. We added footnote 5 addressing this point in Line 431.
Q2: How much of LMCircuit's improvement comes from its zero-cost classifier versus adaptive querying?
The standalone accuracy of LMCircuit's zero-cost classifier corresponds to the leftmost point of each frontier in Fig. 4: 79.85% on BeaverTails and 76.53% on PolitiCAUSE. To isolate the benefit of adaptive querying, we compare how much LMCircuit must spend to achieve the same target accuracy against two non-adaptive baselines.
Supervised stacker. We train a text-only logistic regression on the prompt embeddings, followed by a separate logistic regression for each fixed subset of classifiers. The second model combines the text-only prediction with the selected classifiers' answers.
Non-adaptive circuit. We use LMCircuit's own learned belief model to combine the answers of each fixed subset of classifiers. This baseline shares the same text-only prediction, belief model, and calibration as LMCircuit. The only difference is that it queries a fixed subset, whereas LMCircuit chooses which classifiers to query and when to stop based on the prompt and previously observed answers.
For each method, we select the lowest-cost operating point that achieves a given target accuracy on the validation set, considering all 2∣H∣ fixed subsets for the baselines. We then report the resulting test costs and LMCircuit's percentage savings over each baseline, given by 100(1−CLM/Cbaseline)% (higher is better). Positive values indicate cost reductions (e.g., 50% means LMCircuit uses half the baseline's cost), while negative values indicate higher costs.
BeaverTails
PolitiCAUSE
Across both datasets, LMCircuit achieves substantial cost savings at most target accuracies. On BeaverTails, it reduces query costs by 20–71% relative to supervised stacking and 52–83% relative to the non-adaptive circuit. On PolitiCAUSE, the corresponding savings are 49–69% and 44–60% for targets of 79–81%. At the lowest PolitiCAUSE target, the stacker is cheaper. The cost savings therefore demonstrate the benefit of adaptively selecting classifiers and deciding when to stop. We added these results to Section C.4.4 (Line 1538) in the appendix.