- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce AURA, a method that estimates the benefit of repeated LLM calls for answer correctness purely from the outputs, by modeling these calls as draws from a distribution over answer categories. Evaluation on synthetic data and math problems shows that AURA estimates a smaller budget than baseline methods, but that this improvement does not lead to any significant increase in accuracy or efficiency.
Critical strengths
The paper tackles an important question, which is to reduce the cost of repeated LLM calls in question-answering by estimating when further probing stop being informative about the correct answer.
Critical weaknesses
The paper has several weaknesses, of which the most critical are outlined below
Clarity.
Even with careful reading, the paper is nearly unparseable. The authors have relied heavily on LLM-generated writing, which has resulted in extreme brevity and unnecessary detail (particularly in numbers). Examples of this include the following:
- An extremely jargon- and number-heavy abstract that is not framed well (e.g., the contrastive framing in 'The resulting VOTE-ESS is an operational matched-budget index, not an estimator of latent correlation.').
- The introduction opening as a dense literature review with semicolon-separated clauses, rather than a proper motivation or problem statement.
- An extreme use of poorly-defined jargon throughout the paper (e.g., in section 5.2 with no further explanation: 'The mechanisms are a trajectory-specific Dirichlet law, balanced replication of independent vote prototypes, Markov copy/redraw, and a shared-batch answer mixture.'
Significance
The paper's central result does not actually appear to improve on simply conducting a pre-specified number of calls (e.g., 64) by a significant amount.
- There is no proof that the estimator used is consistent, but extensive experiments are only on synthetic data. Experiments with widely used benchmarks only use Qwen/DeepSeek models. Either a proof or additional experiments on realistic settings would go a long way.
- Table 5 shows that the plurality-utility gap that the paper outlines is under 10% in every case. Table 16 appears to show that these results are not significant.
- Table 6 shows that AURA only leads to the Qwen calls requiring ~52 fewer tokens on average, which is a nearly negligible difference across multiple model calls for any task requiring multiple long-form responses.
Questions for authors
My assessment primarily hinges on the readability of the work, and thus is unlikely to be affected by the following questions. However, the key questions I have are the following:
- In your own words, what are the benefits of using AURA over a fixed budget? How significant are the improvements (particularly the savings) in a real-world context?
- Can you elaborate on your choice of baselines, and how does this compare to the standard measures of cross-answer correlation (semantic similarity, changes in pass@k rates, etc.)?
My overall recommendation is that the authors rewrite the papers in their own words or with the assistance of a human writer.
3 comments
Keeping my score
Thank you for your response. The writing still needs refinement, and your response mostly agrees with my review. You state that your gains are 0.27 calls or 58.5 tokens per question, which is not a significant enough result to merit the use of AURA in my view. You also state that AURA has no benefit as a stopping rule.
Thus, I will be keeping my score, and wish you the best for future revisions.
How often did the stopping rule change the outcome?
Thank you for pressing us on whether the savings justify changing how we stop. On Qwen, the learned rule saved 0.27 calls and 58.5 tokens per question compared with stopping by raw call count. Voting accuracy fell by 0.11 percentage points, and the 95% interval for the combined accuracy and cost measure included zero (Table 6, p. 8).
We also looked at each of the 999 questions separately, averaging over its recorded appearances. On 324 questions the learned rule used fewer calls with the same voting accuracy. On two it used fewer calls and accuracy fell. None of the 999 had higher voting accuracy under the learned rule.
The seven-call example on p. 1 explains what AURA measures. Suppose three calls return the correct answer. If the other four agree on one wrong answer, that answer wins the vote four to three. If the four split evenly between two wrong answers, the correct answer wins. AURA compares the accuracy of these votes with what we would expect from equally many independent draws from a matched reference. We agree that the small savings we observed give little practical reason to replace the simpler stopping rule.
Cross-fit correction, clarity, significance, fixed budgets, baselines
Thank you for the review. Page numbers refer to the revised PDF.
A correction first. The submitted exact-condition cross-fit (Table 5) matched runs by question identifier, but some run families reused identifiers for different questions: 630 of its 752 test trajectories had a reference built entirely from answers to another question. Matching on the question asked (its text and accepted answer) leaves 451 trajectories on 155 distinct questions. Their terminal reference-minus-observed plurality-utility gap is −0.0012 (95% question-bootstrap interval [−0.0113, +0.0084]), or −0.12 percentage points (pp); for the 227 trajectories (95 questions) labelled test in their own run it is +0.0065 [−0.0055, +0.0194]. We withdraw the submitted +0.0011 [−0.0075, +0.0103] and the positive MATH-500 64-call slice (+0.0920 [+0.0195, +0.1825], 40 questions; now −0.0224 [−0.0837, +0.0239]); see Table 5 (p. 7) and Appendix M (pp. 41–42).
Clarity. Agreed. We fixed this only in part. The abstract (p. 1) now gives the question and findings in words, without numbers, and the sentence you quoted is gone. The introduction (p. 1) opens with a seven-call voting example. Section 5.2 (p. 5) gives each simulated mechanism one plain sentence. The opening of Section 6 and the conclusion (pp. 8–9) are rewritten. Sections 2–5 otherwise changed little and remain dense with numbers. As disclosed on p. 10, generative AI helped draft and revise the manuscript.
Significance.
Q1. As a stopping rule, AURA gives no benefit we can show. Learned and raw-prefix rules minus a fixed budget, Qwen-Max on MATH-500:
Rows are re-scored to accept answers equal to the gold up to formatting (Appendix B.1, p. 17). The two rules stopped identically on 40 of 40 questions (L.3) and on 127 of 128 in each new run: the saving against 64 calls is early stopping as such. The learned budget adds only the Table 6 saving, and in simulation it raises stopping regret in all four mechanisms (+0.0324 [+0.0293, +0.0355] at 0.01 per call; Section 5.2, p. 6). AURA's use is measurement: do M recorded calls vote as well as M independent draws from a matched distribution?
Q2. We did not compare with semantic similarity or pass@k. For stopping, the raw-prefix rule isolates the budget (Section 3.4, p. 4). Prespecified measurement baselines are raw count, distinct-answer count and collision ESS, n/[1 + (n − 1)ρ] with ρ an exact-match cross-answer correlation (Appendix J.1, p. 29). In simulation VOTE-ESS beats collision ESS against the (interpolating) vote-equivalent oracle but not against the known count of independent draws in block replay (Appendix L.6, p. 38). Semantic equivalence enters only as merged answer forms (Section 3.1, p. 2), and open-ended outputs are untested (Appendix I, p. 27). Pass@k asks whether any answer is right, which matters with a verifier. In the seven-call example it is 1 either way while the vote differs.