acceptodds
Under review as a conference paper at ICLR 2027

AmBR: Agreement-Based Routing for Efficient Reasoning with Large Language Models

Abstract

Large language models demonstrate strong reasoning capabilities, but deploying them at scale involves a major cost-accuracy trade-off between inexpensive but weaker models and highly capable yet costly ones. We introduce Agreement-Based Routing (AmBR), a lightweight yet effective routing strategy that uses inter-model agreement as a proxy for answer correctness. AmBR first queries two inexpensive models and returns their shared prediction when they agree, escalating only disagreements to a larger (more expensive) fallback model, requiring no training, additional sampling, or model modifications. We evaluate AmBR across multiple reasoning benchmarks using a diverse set of open and closed models. Our results show that agreement between different models is a highly reliable and low-cost correctness signal, enabling substantial reductions in inference cost compared with strong baselines and prior works, while maintaining competitive or even better accuracy, demonstrating a practical and model-agnostic pathway to scalable reasoning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
efficient LLM inference, cost-accuracy trade-off, LLM reasoning, training-free inference, mathematical reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_kap1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a simple workflow that leverages the agreement of two small LLMs to inform a larger model and produce the correct solution to a problem. While agreement does not incur any additional cost, disagreement requires the larger model to reconcile two opposing views/results and produce the final answer. The paper shows that this framework trades a small decrease in performance (top@k) with higher token-efficiency on several standard benchmarks. Theoretically, the paper models agreement between two models and shows that it boosts precision (under independent error, something I come back to later in this review).

Critical strengths

The paper is well written and easy to follow, with a lot of experiments and a theoretical methodology that is not just an ornament. I appreciate that the authors formalise the notion of uncorrelated error, and further expand it to encompass the case the errors are correlated. Results show that one can trade performance for token-efficiency by first checking if two small models agree and give the same solution for a problem.

Critical weaknesses

(major) I find it difficult to find the novelty in the approach; on the one hand, it is not true that different LLMs have uncorrelated errors, as other works [1] and your experiments (see Figure 2) seem to suggest the opposite. I do not really understand what the authors mean by “errors are idiosyncratic rather than correlated”; the phrasing is evocative but the authors should define such a concept. My understanding is that the paper only provides empirical evidence (Figure 2) on classic datasets that are, in some cases, known to be memorised by these models (see the famous case of Qwen2.5 and GSM8K). Also, the literature that picks different models to solve the same task is very rich and dates back to the 2000 (ensembles, weak learners, etc.), with some recent papers that have been impactful in the LLM-agents community [2].

(major) Two costs the authors did not consider are FLOPs and the amount of compute needed: Ambr always employs the largest model it is compared to, plus two small models, resulting in higher compute costs (for example, one Llama70b requires between one and two high-end GPUs at full capacity; adding two small models requires one to two additional GPUs). Also, the inference cost does not only depend on the input length and generated tokens (what people denote as T), but also on the hidden dimensions and, in general, the execution time (which, in your case, would increase if you do not amortise the cost by leveraging asynchronicity via distributed GPUs and/or tools like vLLM). Also, the accuracy reported by Ambr is almost always lower than the base model (Table 1 Llama 3.3 70B (L70) vs AmBR 1–4 avg (L70)), so I do not see the advantage of using this method over a single model. Also, the authors did not report how many samples they used in their experiments, so it is very hard for me to judge if the results are representative (there are no standard deviations or confidence intervals; a McNemar test would suffice here).

(major) Another concern comes from the theoretical aspects: Proposition 3.1 relies on very strong assumptions on how different models behave. Consider Appendix I: in general, it is not true that A = (C1 ∩ C2) ∪ (A ∩ ¬C1 ∩ ¬C2), unless you assume independence between C1 and C2. In fact, for most tasks, models correlate and so do their errors. On the other hand, the general statement (that the authors do in the main paper) relies on computing ρc, which is intractable as it depends on the distribution of two LLMs over, potentially, any input prompt that expresses a problem.

(minor) While the paper is not on benchmarking (that’s why I consider this “minor”), the models considered are quite outdated, especially Gemma2 and Llama3 (the current Gemma is at version 4, while Llama3 is from 2023). As the paper does not fine-tune any model, I do not understand why they did not pick recent models, as that would benefit the applicability of their results (and also benefit from an efficient mixture of experts in the architecture).

[1] Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), NeurIPS 2025 [2] More Agents is All you Need, TMLR

Questions for authors

Q1. Why did the authors choose old models to prove their results hold?

Q2. How do the authors give evidence that the tradeoff between token-efficiency and performance on a task is in favour of their method, since results in Table 1 show that the accuracy of their ensemble is lower than that of a single model?

Q3. How do the authors compute ρc efficiently? That value depends on the entire distribution of the questions that express the same problem, so it is intractable: the independence assumption allows us to skip that calculation and avoid that problem.

Reviewer_9sPaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a training-free LLM routing strategy: for a given query, two small LLMs first generate candidate answers independently. If the two answers agree, the agreed answer is returned as the final answer; otherwise, the query is routed to a larger fallback model.

Critical strengths
  1. Easy to follow.
  2. Some theoretical analysis and some insight why this simple method works.
Critical weaknesses
  1. The term "cost" is never defined. It is used throughout, especially in Tables 1 and 2, but could refer either to computation (FLOPs, tokens, wall-clock time) or to money spent on API or serving calls. Since the method's value depends on which is meant, the paper should define cost explicitly, report both where possible, and state how the small-model calls and the fallback call are counted.

  2. The worst case is slower and more expensive than simply using the large model. When the two small models disagree, both a small and a large model are run on the same query. The paper should report the latency distribution.

  3. Fallback rates should be broken down by dataset. A single aggregate figure hides how often the small models agree, which differs across tasks and difficulty levels. Per-dataset rates would tie the routing behaviour to the reported accuracy and cost.

  4. The method is demonstrated only on tasks whose answers are easy to verify. For multi-step or agentic tasks, where the output is a trajectory rather than a single answer, it is unclear what agreement means or how the method would proceed. The authors should either test such a setting or state the conditions under which the method applies.

Questions for authors

See Critical Weaknesses

Reviewer_hrGhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Agreement-Based Routing (AmBR), a training-free framework for improving the cost–accuracy trade-off in LLM reasoning. AmBR queries two lower-cost models and directly accepts their prediction when the final answers agree, while routing disagreements to a stronger fallback model. The authors provide a theoretical analysis of agreement as a correctness signal, emphasizing the benefits of model diversity and reduced error correlation. Experiments across multiple mathematical and natural language reasoning benchmarks demonstrate that AmBR substantially reduces inference costs while maintaining competitive accuracy, achieving over 60% cost savings in the commercial setting. Comparisons with random routing, learned routing, agreement-based cascading, and sampling-based methods, together with analyses of cross-model agreement and robustness, further support the effectiveness of the proposed approach.

Critical strengths

none

Critical weaknesses

[W1] Limited generalizability of answer agreement as a correctness signal. AmBR assumes that agreement between two models' final answers reliably indicates correctness. Although theoretically motivated, this assumption depends on weakly correlated model errors, which may not hold in practice. The effectiveness demonstrated on selected benchmarks does not necessarily generalize to other domains, particularly when models share systematic errors or produce the same incorrect answers.

[W2] Neglecting reasoning-process quality beyond final-answer accuracy. AmBR relies solely on final-answer agreement for routing decisions, overlooking differences in reasoning quality. Even when two models produce the same correct answer, their reasoning paths may differ substantially in validity and usefulness. This limitation is particularly relevant to reasoning-data synthesis and distillation, where stronger teacher models may still be necessary to ensure high-quality reasoning traces and downstream training benefits.

Questions for authors

none

Reviewer_niTwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Agreement-Based Routing (AmBR): for each question, two low-cost language models are first called; if the answers match, the answer is accepted directly; otherwise, a stronger model is called. Standardized matching is used for numerical and multiple-choice questions, symbolic equivalence judgment is used for mathematical expressions, and bidirectional natural language inference is used to determine semantic consistency for free text. The paper validates this strategy under various inference tasks, model combinations, and perturbation settings, and analyzes the consistency between models and the consistency of multiple samples from the same model. Key results show that this method can achieve a reduction in inference costs at the expense of some accuracy; commercial model configurations also report an improvement in average accuracy.

Critical strengths
  • The problem has clear practical value, and the method is easy to deploy. Routing relies solely on model output, requiring no access to the model's internal state or the training of a dedicated router for each task.

  • The experimental coverage is comprehensive. In addition to the multi-benchmark and multi-model results in the main text, the appendix includes experiments with input perturbations, different prompting methods, model pairing, MATH difficulty stratification, and three random seeds. These experiments support a usable cost-accuracy trade-off across various settings.

  • Some gains are practically significant. In Table 1, the average accuracy of AmBR₂ is 81.2, corresponding to 82.4 for L70, while the cost decreases from 384 to 247. In Table 8, the accuracy of AmBR Pro improves from 87.8 for Gemini to 88.9, while the cost decreases from 4355 to 1708.

  • The paper does not completely evade limitations. The authors discuss the impact of relevant errors, small answer space, and relative model cost; Appendix H also explicitly states that insignificant differences cannot be interpreted as statistical equivalence.

Critical weaknesses

W1. The differences from existing Agreement Based Cascading (ABC) are insufficient to establish a clear methodological innovation.

  • The authors have discussed ABC, but mainly attribute the differences to fixing two models, not adjusting the threshold, and performing only one upgrade. Algorithmically, ABC already allows for two-level cascading and configurable pre-level integration. Setting the pre-level to two models and accepting when they agree and upgrading when they disagree yields the core decision rule of AmBR. Therefore, "no threshold adjustment required" is more like a fixed configuration choice and does not yet reflect a new routing mechanism. This is based on a direct comparison of the two rules, not just a similar name or intuition.

  • Table 2 further fixes the pre-level of ABC to four models, while AmBR only calls two. Therefore, the cost advantage is also affected by the number of pre-level models and cannot be entirely attributed to the method itself. The original ABC paper also studied different integration scales and pointed out that two to three models often provide a good compromise.

  • In addition, Semantic Agreement Enables Efficient Open-Ended LLM Cascades (Soiffer et al., EMNLP 2025) has explored training-free cascading using the semantic consistency of heterogeneous models, but this paper does not discuss this directly related work. More importantly, this paper needs to clarify whether the new knowledge, relative to the above methods, is a new mechanism or a more systematic empirical understanding of a specific configuration.

W2. Existing experiments have not adequately isolated the contribution of "model diversity".

  • Table 4 compares the agreement precision of same-model and cross-model pairings, but the coverage and underlying model capabilities differ across configurations, making it impossible to directly judge the merits of the complete routing system based on this alone.

  • The most crucial comparison should be: Sampling the same small model independently twice; if consistent, accept; otherwise, invoke the same fallback as AmBR.

  • This comparison should be reported alongside the cross-model approach for final accuracy, total cost, and coverage. Appendix D.1 provides valuable same-model sampling plus fallback experiments (n=5, 10), but it does not replace the comparison (n=2) directly corresponding to the two preceding calls of AmBR.

  • Furthermore, the results in Table 4 do not individually support the advantages of cross-model pairings. For example, the precision of L8+L8 on P2 is 87.0, higher than L8+G9's 83.3. Therefore, the overall trend should be distinguished from per-configuration guarantees. The paper also needs to report sampling settings such as temperature, because these settings directly affect the consistency rate of the outputs of the same model in two separate tests.

W3. The assumptions and conclusions of the theoretical analysis need to be more rigorously aligned.

  • The proof of Proposition 3.1 relies on the independence of correct events, but the two conditions listed in the proposition do not explicitly include this assumption, although it is mentioned in the title and context. More importantly, Appendix I directly uses the product of overall accuracy p1p2 from "predictions are independent given the true labels," which is generally not valid: when the accuracy of each class is different, it should be conditionalized by class before averaging.

  • Furthermore, the consensus set precision bound given in Eq. (6) does not unconditionally guarantee that the accuracy is higher than the stronger of the two single models. For example, under independent binary classification errors, if p1=0.9, p2=0.1, the consensus set precision is only 0.5. Therefore, the statement "agreement boosts precision" needs to have clearly defined applicable conditions.

W4. Cost metrics in the free-text experiments are insufficient to demonstrate actual efficiency advantages.

  • The main experiment already uses token fees, which is reasonable; however, the "Cost Advantage" in Table 3 is actually defined as the proportion of samples that do not invoke fallback. The 57% for AmBR and 40% for Hybrid LLM are therefore coverage comparisons, not net cost savings.

  • AmBR requires running two generative models for each question and performing bidirectional NLI; Hybrid LLM has a different invocation path. These overheads need to be factored in to draw cost conclusions in the free-text scenario.

  • Furthermore, the latency analysis in Appendix G.2 uses vendor throughput and assumes that each model generates the same number of tokens. This can serve as an illustrative analysis but is insufficient to support a representation of measured speeds or worst-case latency bounds.

W5. The statement "maintaining accuracy" is stronger than existing results, and some replicate results require explanation.

  • A smaller average loss does not mean that every task is close to a fallback. For example, on SQA− in Table 1, AmBR₄ is 75.3 and L70 is 81.7, a difference of 6.4 percentage points; similar discrepancies are shown in the three sub-results of Table 19. Therefore, the conclusion should be stated as "reducing costs at a smaller average accuracy cost in some configurations," rather than vaguely emphasizing no sacrifice in reliability.

  • Furthermore, some differences between single experiments and the three sub-results warrant clarification: on SciNLI, L8 in Table 1 is 49.8, while in Table 19 it is 44.59+-0.79; M8 is 43.3 and 48.97+-0.23 respectively. It requires explanation for whether there are differences in hints, decoding, model versions, or evaluation protocols.

Questions for authors
  1. When using the same two preceding models, the same fallback, and a consistent acceptance rule, what are the specific differences in the prediction and invocation paths between AmBR and ABC? If these differences don't exist, how should the independent contribution of this paper be repositioned?

  2. Please supplement the end-to-end results for “two samples from the same model + the same fallback” using the existing sampling results, and report the complete decoding setup.

  3. Please clarify the source of randomness in Proposition 3.1, the independence assumption, and the conditions under which the method improves accuracy over a single model.

  4. After accounting for the costs of two generative models and bidirectional NLI, does the efficiency advantage of Table 3 still exist? What causes the difference in SciNLI between Table 1 and Table 19?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.