- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a simple workflow that leverages the agreement of two small LLMs to inform a larger model and produce the correct solution to a problem. While agreement does not incur any additional cost, disagreement requires the larger model to reconcile two opposing views/results and produce the final answer. The paper shows that this framework trades a small decrease in performance (top@k) with higher token-efficiency on several standard benchmarks. Theoretically, the paper models agreement between two models and shows that it boosts precision (under independent error, something I come back to later in this review).
Critical strengths
The paper is well written and easy to follow, with a lot of experiments and a theoretical methodology that is not just an ornament. I appreciate that the authors formalise the notion of uncorrelated error, and further expand it to encompass the case the errors are correlated. Results show that one can trade performance for token-efficiency by first checking if two small models agree and give the same solution for a problem.
Critical weaknesses
(major) I find it difficult to find the novelty in the approach; on the one hand, it is not true that different LLMs have uncorrelated errors, as other works [1] and your experiments (see Figure 2) seem to suggest the opposite. I do not really understand what the authors mean by “errors are idiosyncratic rather than correlated”; the phrasing is evocative but the authors should define such a concept. My understanding is that the paper only provides empirical evidence (Figure 2) on classic datasets that are, in some cases, known to be memorised by these models (see the famous case of Qwen2.5 and GSM8K). Also, the literature that picks different models to solve the same task is very rich and dates back to the 2000 (ensembles, weak learners, etc.), with some recent papers that have been impactful in the LLM-agents community [2].
(major) Two costs the authors did not consider are FLOPs and the amount of compute needed: Ambr always employs the largest model it is compared to, plus two small models, resulting in higher compute costs (for example, one Llama70b requires between one and two high-end GPUs at full capacity; adding two small models requires one to two additional GPUs). Also, the inference cost does not only depend on the input length and generated tokens (what people denote as T), but also on the hidden dimensions and, in general, the execution time (which, in your case, would increase if you do not amortise the cost by leveraging asynchronicity via distributed GPUs and/or tools like vLLM). Also, the accuracy reported by Ambr is almost always lower than the base model (Table 1 Llama 3.3 70B (L70) vs AmBR 1–4 avg (L70)), so I do not see the advantage of using this method over a single model. Also, the authors did not report how many samples they used in their experiments, so it is very hard for me to judge if the results are representative (there are no standard deviations or confidence intervals; a McNemar test would suffice here).
(major) Another concern comes from the theoretical aspects: Proposition 3.1 relies on very strong assumptions on how different models behave. Consider Appendix I: in general, it is not true that A = (C1 ∩ C2) ∪ (A ∩ ¬C1 ∩ ¬C2), unless you assume independence between C1 and C2. In fact, for most tasks, models correlate and so do their errors. On the other hand, the general statement (that the authors do in the main paper) relies on computing ρc, which is intractable as it depends on the distribution of two LLMs over, potentially, any input prompt that expresses a problem.
(minor) While the paper is not on benchmarking (that’s why I consider this “minor”), the models considered are quite outdated, especially Gemma2 and Llama3 (the current Gemma is at version 4, while Llama3 is from 2023). As the paper does not fine-tune any model, I do not understand why they did not pick recent models, as that would benefit the applicability of their results (and also benefit from an efficient mixture of experts in the architecture).
[1] Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond), NeurIPS 2025 [2] More Agents is All you Need, TMLR
Questions for authors
Q1. Why did the authors choose old models to prove their results hold?
Q2. How do the authors give evidence that the tradeoff between token-efficiency and performance on a task is in favour of their method, since results in Table 1 show that the accuracy of their ensemble is lower than that of a single model?
Q3. How do the authors compute ρc efficiently? That value depends on the entire distribution of the questions that express the same problem, so it is intractable: the independence assumption allows us to skip that calculation and avoid that problem.