acceptodds
Under review as a conference paper at ICLR 2027

AURA: Measuring Evidence from Repeated Calls to Hosted Foundation Models

Abstract

Repeated sampling is a standard way to improve foundation-model answers, but on a hosted service each sample is a billed API call. We introduce AURA, an output-only framework for asking how many independent categorical votes would match the plurality utility of a repeated-call trajectory. The resulting VOTE-ESS is an operational matched-budget index, not an estimator of latent correlation. We validate it on 180 Dirichlet–multinomial configurations and an exact block-replication control: it decreases with dependence in every fixed configuration and recovers the block budget exactly in 96.3% of cases. In a broader 288-cell stress test spanning four dependence mechanisms, an interpolated isotonic categorical VOTE-ESS reduces non-i.i.d measurement error by 86.4% relative to the strongest output-only proxy. Its exact-block compatible intervals, however, cover only 83.5%, below a prespecified 90% coverage threshold, and its stopping controller incurs 0.0324 more regret than a raw-prefix controller. On frozen Qwen trajectories, the categorical raw/isotonic terminal ratios are 0.944/0.981 over 2,052 disjoint test trajectories, while the direct reference-minus-observed plurality gap is (95% question-bootstrap CI ). An exact-condition cross-fit over 752 test trajectories instead gives . Breadth checks over 69 aliases and full matched runs on two official DeepSeek models show that the protocol transfers, but not that all providers are independent. Finally, prefix-only controllers save about 0.22–0.25 calls without detectable gains in accuracy or matched objective. These results support a disciplined way to measure repeated-call evidence while showing that measurable shrinkage need not justify a different test-time compute policy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
hosted foundation models, black-box evaluation, repeated sampling, test-time compute, self-consistency, effective sample size, plurality voting, adaptive stopping, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_yAhoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce AURA, a method that estimates the benefit of repeated LLM calls for answer correctness purely from the outputs, by modeling these calls as draws from a distribution over answer categories. Evaluation on synthetic data and math problems shows that AURA estimates a smaller budget than baseline methods, but that this improvement does not lead to any significant increase in accuracy or efficiency.

Critical strengths

The paper tackles an important question, which is to reduce the cost of repeated LLM calls in question-answering by estimating when further probing stop being informative about the correct answer.

Critical weaknesses

The paper has several weaknesses, of which the most critical are outlined below

Clarity.

Even with careful reading, the paper is nearly unparseable. The authors have relied heavily on LLM-generated writing, which has resulted in extreme brevity and unnecessary detail (particularly in numbers). Examples of this include the following:

  • An extremely jargon- and number-heavy abstract that is not framed well (e.g., the contrastive framing in 'The resulting VOTE-ESS is an operational matched-budget index, not an estimator of latent correlation.').
  • The introduction opening as a dense literature review with semicolon-separated clauses, rather than a proper motivation or problem statement.
  • An extreme use of poorly-defined jargon throughout the paper (e.g., in section 5.2 with no further explanation: 'The mechanisms are a trajectory-specific Dirichlet law, balanced replication of independent vote prototypes, Markov copy/redraw, and a shared-batch answer mixture.'

Significance

The paper's central result does not actually appear to improve on simply conducting a pre-specified number of calls (e.g., 64) by a significant amount.

  • There is no proof that the estimator used is consistent, but extensive experiments are only on synthetic data. Experiments with widely used benchmarks only use Qwen/DeepSeek models. Either a proof or additional experiments on realistic settings would go a long way.
  • Table 5 shows that the plurality-utility gap that the paper outlines is under 10% in every case. Table 16 appears to show that these results are not significant.
  • Table 6 shows that AURA only leads to the Qwen calls requiring ~52 fewer tokens on average, which is a nearly negligible difference across multiple model calls for any task requiring multiple long-form responses.
Questions for authors

My assessment primarily hinges on the readability of the work, and thus is unlikely to be affected by the following questions. However, the key questions I have are the following:

  • In your own words, what are the benefits of using AURA over a fixed budget? How significant are the improvements (particularly the savings) in a real-world context?
  • Can you elaborate on your choice of baselines, and how does this compare to the standard measures of cross-answer correlation (semantic similarity, changes in pass@k rates, etc.)?

My overall recommendation is that the authors rewrite the papers in their own words or with the assistance of a human writer.

3 comments
AuthorsOpenReview ↗

Cross-fit correction, clarity, significance, fixed budgets, baselines

Thank you for the review. Page numbers refer to the revised PDF.

A correction first. The submitted exact-condition cross-fit (Table 5) matched runs by question identifier, but some run families reused identifiers for different questions: 630 of its 752 test trajectories had a reference built entirely from answers to another question. Matching on the question asked (its text and accepted answer) leaves 451 trajectories on 155 distinct questions. Their terminal reference-minus-observed plurality-utility gap is −0.0012 (95% question-bootstrap interval [−0.0113, +0.0084]), or −0.12 percentage points (pp); for the 227 trajectories (95 questions) labelled test in their own run it is +0.0065 [−0.0055, +0.0194]. We withdraw the submitted +0.0011 [−0.0075, +0.0103] and the positive MATH-500 64-call slice (+0.0920 [+0.0195, +0.1825], 40 questions; now −0.0224 [−0.0837, +0.0239]); see Table 5 (p. 7) and Appendix M (pp. 41–42).

Clarity. Agreed. We fixed this only in part. The abstract (p. 1) now gives the question and findings in words, without numbers, and the sentence you quoted is gone. The introduction (p. 1) opens with a seven-call voting example. Section 5.2 (p. 5) gives each simulated mechanism one plain sentence. The opening of Section 6 and the conclusion (pp. 8–9) are rewritten. Sections 2–5 otherwise changed little and remain dense with numbers. As disclosed on p. 10, generative AI helped draft and revise the manuscript.

Significance.

  • A fixed budget. Agreed; see Q1.
  • Proof and providers. Correct: there is no proof, and all hosted data come from Qwen/DashScope and DeepSeek (Section 6, p. 9). Consistency would rest on independent draws from a stable answer distribution, which our runs do not support: responses are not exchangeable across runs (permutation p = 0.046; Appendix M, p. 42). The discrete VOTE-ESS inverse is also inconsistent where the reference curve is flat at the observed utility (Section 3.2, p. 3), and the stress-test intervals under-cover (Table 3, p. 6).
  • Tables 5 and 16. Agreed. All gaps in the corrected Table 5 lie within 3 percentage points of zero, no cross-fit interval excludes zero, and none of Table 16's three unadjusted positive intervals (69 aliases, pp. 22–23) survives Holm adjustment. This is not evidence of independence: under simulated independence the gap averages −0.0054 (s.d. 0.0036), below the observed −0.0012, and four-fold copying of every answer is detected in only 38% of simulations by this gap, though in 100% by the one-to-four gain gap (p. 42). Plug-in rows, Table 16 included, estimate the reference from the evaluated answers and are descriptive only (pp. 41–42).
  • Table 6. Agreed: over the raw-prefix rule (the same rule on raw call count), the learned rule saves 0.27 calls [0.23, 0.30] and 58.5 tokens [49.3, 67.7] per question, and changes accuracy by −0.11 pp [−0.31, 0.00] and the matched objective by +0.0016 [−0.0006, +0.0029] (p. 8), clustering on 999 distinct questions.

Q1. As a stopping rule, AURA gives no benefit we can show. Learned and raw-prefix rules minus a fixed budget, Qwen-Max on MATH-500:

Fixed budget; questionsCallsPlurality utility, pp [paired 95% CI]
64; 40 (L.3, p. 35)3.7 vs 640.0 [−7.5, +7.5] (archived scorer −2.5)
4; 128 held-out, 64-call run (p. 36)3.6 vs 4−2.93 [−5.08, −0.98] (archived −2.15)
64; same 128, 64-call run (p. 36)3.6 vs 64−5.47 [−10.94, −0.78] (archived −1.56)
4; same 128, earlier 32-call run (p. 36)3.8 vs 4+0.59 [−1.17, +2.54] (archived +0.20)

Rows are re-scored to accept answers equal to the gold up to formatting (Appendix B.1, p. 17). The two rules stopped identically on 40 of 40 questions (L.3) and on 127 of 128 in each new run: the saving against 64 calls is early stopping as such. The learned budget adds only the Table 6 saving, and in simulation it raises stopping regret in all four mechanisms (+0.0324 [+0.0293, +0.0355] at 0.01 per call; Section 5.2, p. 6). AURA's use is measurement: do M recorded calls vote as well as M independent draws from a matched distribution?

Q2. We did not compare with semantic similarity or pass@k. For stopping, the raw-prefix rule isolates the budget (Section 3.4, p. 4). Prespecified measurement baselines are raw count, distinct-answer count and collision ESS, n/[1 + (n − 1)ρ] with ρ an exact-match cross-answer correlation (Appendix J.1, p. 29). In simulation VOTE-ESS beats collision ESS against the (interpolating) vote-equivalent oracle but not against the known count of independent draws in block replay (Appendix L.6, p. 38). Semantic equivalence enters only as merged answer forms (Section 3.1, p. 2), and open-ended outputs are untested (Appendix I, p. 27). Pass@k asks whether any answer is right, which matters with a verifier. In the seven-call example it is 1 either way while the vote differs.

Reviewer_yAhoOpenReview ↗

Keeping my score

Thank you for your response. The writing still needs refinement, and your response mostly agrees with my review. You state that your gains are 0.27 calls or 58.5 tokens per question, which is not a significant enough result to merit the use of AURA in my view. You also state that AURA has no benefit as a stopping rule.

Thus, I will be keeping my score, and wish you the best for future revisions.

AuthorsOpenReview ↗

How often did the stopping rule change the outcome?

Thank you for pressing us on whether the savings justify changing how we stop. On Qwen, the learned rule saved 0.27 calls and 58.5 tokens per question compared with stopping by raw call count. Voting accuracy fell by 0.11 percentage points, and the 95% interval for the combined accuracy and cost measure included zero (Table 6, p. 8).

We also looked at each of the 999 questions separately, averaging over its recorded appearances. On 324 questions the learned rule used fewer calls with the same voting accuracy. On two it used fewer calls and accuracy fell. None of the 999 had higher voting accuracy under the learned rule.

The seven-call example on p. 1 explains what AURA measures. Suppose three calls return the correct answer. If the other four agree on one wrong answer, that answer wins the vote four to three. If the four split evenly between two wrong answers, the correct answer wins. AURA compares the accuracy of these votes with what we would expect from equally many independent draws from a matched reference. We agree that the small savings we observed give little practical reason to replace the simpler stopping rule.

Reviewer_7nrVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper investigates whether the same number of repeated API calls provide the same plurality-voting utility as same number independent draws from a matched categorical answer distribution. They validation via dependence simulations, block-replication experiments, and hosted-model evals on Qwen and Deepseek.

Critical strengths
  1. Their categorical reference preserves competition among distinct wrong answers, which is important for plurality voting, and thus, is a more approriate reference for the rule being studied.
  2. The evaluation distinctly separates the measurement quality from downstream decision value.
Critical weaknesses
  1. The interpretation of VOTE-ESS as an evidence budget is not well validated and is insufficient.
  2. The positive validation does not establish clearly the added value of their proposed index. Althouth this was acknowledged, the paper does not connect the validation across these objects or demonstrate what the index contributes beyond the highlighted gap. Given the absence of a demonstrated downstrream benefit, it leaves the methodological contribution insufficiently established.
  3. I find the paper generally hard to parse due to its dense procedural reporting
Questions for authors

I have no questions that would significantly affect my assessment.

3 comments
AuthorsOpenReview ↗

Corrected cross-fit, limits of VOTE-ESS, and no downstream benefit

Thank you for the review. Page numbers refer to the revised PDF; results without a pointer are not in it.

Correction first. Our exact-condition cross-fit matched runs by question identifier, but run families reused identifiers for different questions: 630 of the 752 test trajectories had a reference built entirely from answers to another question (Appendix M, p. 41). Matching on the question asked (its text and accepted answer) leaves 451 trajectories on 155 questions. Their reference-minus-observed terminal gap in plurality utility is −0.0012, 95% question-bootstrap interval [−0.0113, +0.0084] (−0.12 percentage points, pp), and +0.0065 [−0.0055, +0.0194] on the 227 trajectories (95 questions) labelled test in their own run (Section 5.3, p. 7). Both intervals include zero. Copying every answer four times is detected in only 38% of simulated data sets by this terminal gap but in 100% by the one-to-four gain gap (p. 42); both figures concern copying of every answer, not partial dependence. We withdraw the submitted +0.0011 [−0.0075, +0.0103], the positive MATH-500 64-call slice (40 questions; +0.0920 [+0.0195, +0.1825], now −0.0224 [−0.0837, +0.0239]) and an equal-call allocation replay prepared for W2, whose 140 joined pairs share no question (Appendix M and Table 32, p. 41). Our archived scorer rejected 14.1% of calls equal to the gold up to formatting in two re-scored Qwen-Max collections (Appendix B.1, p. 17).

W1. VOTE-ESS as an evidence budget. We agree; the paper now treats it as secondary to the direct gap (p. 1). In the exact block control it recovers the known number of independent draws in 104 of 108 configurations (96.3%; Section 5.1, p. 4). With finite data (mean absolute error, votes):

TargetEq. (3), pooledEq. (3) per trajectoryInterpolatedCollision ESS
Known draws, finite block control, 108 configurations (pp. 4–5)0.9078.653––
Interpolating oracle, 216 non-i.i.d. cells (pp. 6, 38)0.921–0.1631.206
Known draws, 72 block-replay cells (p. 38)1.253–1.0430.365

Against known draws (last row), collision ESS has about a third of either inverse's error. The 86.4% reduction in Section 5.2 (p. 6) is the middle row. Exact-block interval coverage is 83.5% against a 90% criterion (Table 3, p. 6).

W2. Added value and downstream benefit. We have no downstream benefit to report, and on hosted data we cannot show that the index adds anything to the gap.

Beyond the gap. With u = U_obs(M), Eq. (3) returns M exactly when Δ(M) ≤ 0, so at the terminal budget VOTE-ESS can only re-express a positive gap in calls. Hosted ratios invert each trajectory and then average, the order with 8.653 votes of error above; Sections 3.2–3.3 (p. 3) now state both points. On the 451 verified trajectories VOTE-ESS is at the cap M for 399 (all 280 correct votes among them), at the floor of one for 51, and in between for 1; a wrong vote reaches the floor only if the reference can be right. Floor cases make up 0.109 of the hosted ratio's 0.111 shortfall from one (ratio 0.889, p. 7). Independent votes produce them too, so the shortfall alone does not indicate repeated answers.

Downstream.

  • Simulation: replacing raw prefix length with the predicted budget raises controller regret in all four mechanisms (+0.0324 [+0.0293, +0.0355] overall at 0.01 per call; Section 5.2 and Table 4, pp. 6–7).
  • Hosted (Table 6, p. 8, now clustered on 999 distinct questions): the learned Qwen rule uses 0.27 [0.23, 0.30] fewer calls per question than the raw-prefix rule and changes accuracy by −0.11 pp [−0.31, 0.00] and the matched objective by +0.0016 [−0.0006, +0.0029]; DeepSeek is similar. The two rules stopped at the same call on all 40 questions of the archived 64-call replay and on 127 of 128 in each of two new MATH-500 runs on Qwen-Max (Appendix L.3, p. 35). Re-scored, both rules minus a fixed four-call budget (descriptive) were +0.59 pp [−1.17, +2.54] in the 32-call run and −2.93 pp [−5.08, −0.98] in the 64-call run (+0.20 and −2.15 pp under the archived scorer), and minus fixed 64, −5.47 pp [−10.94, −0.78].
  • Recovery: after a wrong first vote (195 trajectories, 87 questions), final plurality utility averaged 27.3% over questions, against 28.9% for independent votes given the same first vote (gap +1.64 pp [−1.86, +5.86]; Appendix L.5, p. 37); 35 of the 195 ended with a correct plurality, and voting changed the first-call decision in 44 of 451 trajectories. So a dependence-aware budget has no detected deficit to recover; one of up to 5.9 pp is not excluded.

W3. Dense reporting. Partly addressed. The abstract, introduction and conclusion are rewritten around the direct comparison, with a seven-call example (pp. 1, 9). Sections 3–5 remain dense; Section 3.2 now defines the gap (Eq. 2) before VOTE-ESS (Eq. 3) and states their link (p. 3).

Thank you for the response and for reporting the cross-fit error. I am keeping my score at 1.

You agree that VOTE-ESS is not validated as an evidence budget and that it shows no downstream benefit and adds nothing beyond the direct gap on hosted data. The correction also removes the one positive hosted result. What remains is not enough for acceptance.

For a revision, I would build on the one-to-four gain gap, which is the stronger instrument.

AuthorsOpenReview ↗

Comparing the gain from one answer to four

Thank you for identifying the one-to-four gain gap as the stronger direction. We agree that this is the clearer comparison to emphasize.

It compares the improvement from voting in the recorded answers with the improvement expected under the separately collected, question-matched categorical reference:

D1→4=[Uref(4)−Uref(1)]−[Uobs(4)−Uobs(1)].D_{1\to4}=[U_{\mathrm{ref}}(4)-U_{\mathrm{ref}}(1)]-[U_{\mathrm{obs}}(4)-U_{\mathrm{obs}}(1)].

Here, U(m)U(m) is plurality accuracy with fractional credit for ties. The reference retains competition among different wrong answers. Subtracting the one-answer baseline measures the gain from voting relative to each source's single-answer accuracy.

The revised appendix section “Plurality gains across voting budgets” reports the corrected comparison on all 451 matched trajectories covering 155 questions:

QuantityGain, percentage points95% question-bootstrap interval
Recorded answers, one to four votes+1.92[+1.10, +2.79]
Question-matched reference, one to four votes+1.76[+1.06, +2.51]
Reference minus recorded gain−0.16[−0.37, +0.05]

Thus, voting improved accuracy on these recorded questions over a single answer, while the corrected panel does not establish a shortfall relative to the reference. The paired interval in the last row is computed directly, preserving the shared questions; it is not obtained by subtracting the two marginal intervals.

This comparison addresses whether repeated calls realize the voting improvement expected under the specified reference. Its interpretation stays in units of accuracy gain. Calibration as an evidence budget and improved stopping decisions require separate validation, which these results do not provide.

We appreciate your careful assessment and the concrete recommendation to build on this comparison.

Reviewer_qep2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much plurality-voting utility repeated calls to a hosted foundation model provide relative to independent draws from a matched categorical answer distribution. It proposes AURA, which canonicalizes responses, constructs an independent reference curve, and reports both a direct utility gap and VOTE-ESS, an operational matched-budget index rather than an estimate of latent dependence. The evaluation combines controlled dependence simulations with frozen Qwen and DeepSeek trajectories, exact-condition cross-fitting, and prefix-only stopping experiments. In non-i.i.d. simulations, the interpolated categorical estimator reduces measurement error by 86.4% relative to the strongest prespecified output-only proxy, but its block-control compatible intervals achieve only 83.5% coverage. The hosted experiments do not establish an aggregate utility deficit under the evaluated references or improved stopping decisions, making the main contribution a measurement protocol and a bounded negative result rather than a new test-time compute policy.

Critical strengths
  • The authors maintain distinct wrong-answer classes aligns the reference with the actual plurality decision rule, avoiding the information loss caused by collapsing all incorrect answers into one category. The exact block-replication control and four-mechanism stress test provide targeted validation of this design rather than relying solely on answer-agreement statistics.

  • The evaluation carefully separates measurement validity from decision value through exact-condition cross-fitting, question-level uncertainty estimates, and controllers restricted to prefix-visible information. Reporting failed coverage criteria and the absence of stopping gains makes the negative conclusion informative and reduces the risk of interpreting effective-budget shrinkage as an established accuracy loss or practical improvement.

Critical weaknesses
  • The connection between simulation-level validation and trajectory-level budget estimates remains insufficiently established. At the terminal budget, a unique correct plurality gives observed utility one, so Eq. (2) necessarily returns the full budget, whereas Section 5.1 finds that pooling utilities before inversion is essential and Section 5.2 evaluates a different interpolated inverse. Together with the 83.5% block-control interval coverage, these issues leave the reliability and interpretation of the hosted effective-budget summaries less well supported than the synthetic point-estimation results suggest.

  • The hosted-data conclusion remains sensitive to how the categorical reference is estimated. The main plug-in and breadth analyses reuse the evaluated trajectory to construct its reference, while the exact-condition cross-fit covers a smaller, different subset, so differences between these analyses cannot be attributed to reference estimation alone. Matched-subset comparisons and reference-size sensitivity analyses would strengthen the evidence that the reported absence of an aggregate utility deficit is not an artifact of the available reference data.

Questions for authors
  1. Section 5.1 reports substantially greater error when trajectory-wise inverses are averaged rather than utilities pooled, and the stress test uses an inverse distinct from Eq. (2). Could the authors clarify the aggregation order for the hosted budget summaries and compare both inversion conventions on identical data, including finite-reference calibration and interval coverage under independent and block-replay controls?

  2. The exact-condition cross-fit changes both the reference construction and the eligible question set, while the breadth panel contains only one trajectory per exact condition. Could the authors compare plug-in and cross-fitted estimates on the same question–condition subset and vary the number of separately collected reference trajectories to assess the stability of the direct utility gaps and matched-budget estimates?

3 comments
AuthorsOpenReview ↗

Cross-fit correction, budget inversion, reference estimation

Thank you for the valuable feedback! Pages refer to the revised PDF. Gaps are reference minus observed plurality utility in percentage points (pp; −0.12 pp is −0.0012 in the PDF). All new analyses here are post hoc.

Correction. Our cross-fit matched runs by question identifier, which some run families reused for different questions. So 630 of 752 submitted test trajectories had a reference built entirely from answers to another question (120 more partly), and the 752 covered 449 distinct questions, not 160. Matching on the question asked (text and accepted answer) leaves 451 trajectories on 155 questions with a terminal gap of −0.12 pp (95% question-bootstrap CI [−1.13, +0.84]); on the 227 trajectories (95 questions) labelled test in their own run, as in the submitted analysis, it is +0.65 pp [−0.55, +1.94] (Table 5, p. 7; B.4, p. 18). We withdraw the submitted +0.11 pp [−0.75, +1.03] and the positive MATH-500 64-call slice (now −2.24 pp [−8.37, +2.39]). These archived gaps are not re-scored (B.1, p. 17). Details: App. M (pp. 41–42).

W1. Agreed. At the terminal budget Eq. (3) (your Eq. 2) returns M for a correct unique plurality and, for a wrong one, 1 if the reference utility at M is positive and M if it is zero; only ties can fall in between. Of the 451 cross-fit budgets, 399 are at M, 51 at 1 and one in between (§3.2, p. 3). Hosted ratios invert per trajectory, then average, the order with 8.653 votes of block-control error, against 0.907 for pooling first (§3.3, §5.1, pp. 3–5). Hosted ratios are thus not budget estimates; hosted conclusions rest on the direct gap (Eq. 2).

W2. Agreed. The plug-in reference reuses the trajectory's own answers: under simulated independence its gap on 2,052 Qwen test trajectories averages −0.45 pp (observed −0.63 pp), and copying every answer four times moves it by only +0.22 pp, versus +1.16 pp for a reference that excludes the target. So plug-in, breadth and DeepSeek rows are descriptive (Table 5, p. 7). On the cross-fit, the terminal gap detects four copies of every answer in only 38% of 1,000 simulated data sets and two in 10% (test-labelled 14%, 6%), but the one-to-four gain gap detects them in 100% and 70.9% (30.6%, 11.2%), and its observed −0.17 pp lies inside its independence null (App. M, p. 42). Partial dependence is untested; this is not evidence of independence.

Q1. Order: Eq. (3) per trajectory on its raw reference curve, with observed utility (raw, isotonic or cumulative-max) as threshold, then averaged within and across questions (cross-fit) or across trajectories (plug-in; §3.2–3.3, p. 3). On the 5,760 simulation records, reference bank fixed, neither inverse meets the 90% criterion against the archived block oracle (discrete 62.1%, interpolated 83.5%); coverage of known independent and block-replay budgets is 98.9%/69.9% and 93.8%/72.3% (L.2, p. 35; L.6, p. 38). Both inverses are now in the finite-reference simulation (L.4, Table 28a, pp. 36–37). With the reference refitted, discrete intervals contain the known count in 98.8% (independent) and 96.0% (four prototypes) of 400 experiments with 48 reference responses, and 98.8% and 96.8% with 192; the interpolated inverse gives 98.8% and 93.8% at both sizes (with four block reference trajectories: 97.2% vs discrete 95.5%). With a fixed reference, the discrete inverse covers the known block count in only 59.0% (48) and 83.8% (192). Refitted intervals average 8.8 and 12.0 votes on a 16-vote range at 48 responses (6.7 and 7.5 at 192), and 31.5% of block intervals at 48 span 1–16. These invert a utility pooled over 128 targets, not the per-trajectory order behind hosted ratios, and in the independent cells the known count is the 16-vote cap.

Q2. Same targets (Table 27, p. 33):

Plug-inSeparate runsChange [95% CI]
Gap, 451 trajectories−0.46 pp−0.12 pp+0.34 pp [−0.07, +0.84]
Gap, 227 test-labelled−0.37 pp+0.65 pp+1.02 pp [+0.13, +2.13]
Raw ratio, 4510.9110.889−0.022 [−0.041, −0.004]

Subsampling reference pools to K = 32, 16, 8 or 4 responses (317 trajectories with pools ≥ 32) lowers the gap from −0.59 pp to −0.87, −1.34, −1.87 and −2.78 pp, close to the independence expectation at each size (−0.65, −1.10, −1.66, −2.66). Net of it, the gap stays within −0.24 to −0.12 pp (166 test-labelled: +0.39 to +0.48), every interval including zero (L.1, p. 34). A small reference thus masks a deficit rather than producing one, and raises the raw ratio (exact curves: 0.941 at K = 4, 0.863 at full size). On all 227 test-labelled targets the gap exceeds its independence expectation by +1.65 pp [+0.22, +3.32], and by +1.22 to +1.74 pp at K = 4–16, so subsampling does not remove it, but only +0.52 pp [−0.99, +2.11] on the 166 with pools ≥ 32. Responses are not exchangeable across runs (permutation p = 0.012; App. M). Keeping only the nearest run shifts the gap by −0.38 pp [−1.80, +1.01] (L.1).

Reviewer_qep2OpenReview ↗

Response to the Authors

Dear authors,

I appreciate the authors' detailed response, additional analyses, and transparency in identifying and correcting the cross-fitting issue.

However, my main methodological concerns remain. The clarification that hosted VOTE-ESS ratios are computed through per-trajectory inversion, which the authors now acknowledge should not be interpreted as effective-budget estimates, substantially limits the interpretation of the original measurement results. Although the additional finite-reference analyses improve uncertainty coverage under some settings, they do not establish reliable uncertainty quantification for the hosted per-trajectory estimator.

I also appreciate the corrected cross-fitting and matched-subset comparisons. Nevertheless, the original cross-fitting error affects a central validation component, and the revised results remain sensitive to reference construction and sample selection. In particular, the limited statistical power of the terminal utility gap under known block dependence raises further concerns about how strongly the absence of a detected aggregate deficit can be interpreted.

Overall, I acknowledge the value of the controlled simulation results, but I remain uncertain whether the revised empirical evidence sufficiently establishes a reliable and practically meaningful measurement framework for hosted repeated-call trajectories.

AuthorsOpenReview ↗

What the Qwen votes tell us about repeated calls

Thank you for asking what someone measuring repeated calls can actually learn from these results. We agree that the outcome of a single hosted vote cannot tell us how many independent calls it was worth. Across labelled questions, we can ask how often the observed vote succeeds at a given budget and compare that with independent draws from a matched reference.

Your point about calibration holds even with a perfectly known reference. Consider a binary question with a 60% chance of a correct answer per independent call. Draw four answers and repeat each four times. We enumerated every possible vote, giving half credit for a tie. Applied to an individual 16-answer vote, Eq. (3) returns either 1 or 16 and never the known count of 4. Averaging accuracy first gives 64.8%. Three and four independent draws both attain that accuracy, so Eq. (3) selects four by its largest-budget rule (p. 3). Table 28a pools 128 trajectories before inversion. Its coverage therefore leaves uncertainty for a single hosted trajectory unvalidated (p. 37).

We withdrew the positive 64-call MATH-500 finding after discovering that some runs had matched different questions. We checked the corrected Qwen votes one by one. The text above Table 27 (p. 33) gives the wrong simulation count for the plug-in column. It used 32,768 Monte Carlo draws, while the separate-run column used 40,000. With 40,000 on both sides, replacing each trajectory’s own reference with one fitted to separate runs changed the reported count on 26 of 451 trajectories. On 22 it switched between one and the full call budget. With the observed votes held fixed, the average accuracy gap (reference minus observed) shifted by +0.33 percentage points across all 451 trajectories (paired 95% interval −0.08 to +0.83), and by +1.02 points on the 227 labelled test (+0.12 to +2.13). Reference choice matters even after matching the question text and accepted answer.

We share your concern about power. When every answer was copied fourfold, the final-budget gap detected copying in 38% of simulated panels. Comparing the accuracy gain from one to four calls against the matched reference detected the same copying in 100% of full panels and 30.6% on the test-labelled subset. The observed gain gap on the full panel was −0.17 percentage points, inside the range expected under independence (Appendix M, p. 42). We would use these comparisons to evaluate the return on additional calls across a specified set of questions. They leave open both the strength of dependence in the hosted calls and when to stop on an individual question.

Reviewer_dWFWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces a framework (AURA) with a technique to answer the question of when and how repeated calls to the same LLM introduce new information by looking only at the outputs. In the paper, this is formalized by assuming a canonical categorical distribution of answers, and measuring how many iid draws from that distribution would reproduce the effective information in the repeated LLM inference samples. The method introduced relies on the ground truth answers as belonging to a single class of the categorical distribution but different wrong answers come from separate classes. The goal is to establish in simulations and using real LLMs whether the method can identify whether the draws don't provide more info and whether the measure can be used to determine how many draws are enough such that doing more calls more no additional utility. The simulation experiments show that the measure introduced is able to capture effective information in repeated samples but isn't able to provide an effective control, i.e decide how many draws are sufficient. The experiments with real LLMs do not show that the technique is able to answer the original question, (when and how repeated calls introduce new information).

Critical strengths

One strength is in the through construction of the synthetic simulations to evaluate Vote_ESS. While the experiments may not necessarily reflect how real LLM answers would correlate, within the simulation, the papers covers multiple kinds of dependences relationships in Table 4. Table 3 also pre-specifies evaluation criteria and makes it easy to see whether each result met the criteria or not so it was easy to see that compatible coverage, controller regret disadvantage and number of mechanisms where control improves ≥ 10% did not meet the criteria. Overall this shows that the experiments on the synthetic simulations were well designed.

Critical weaknesses

One key weakness is in the presentation, it is a little bit unclear whether \delta (M) is the main metric or whether it's vote-ess. In the introduction, the paper presents vote-ess as the key metric which measures how many actual votes (independent responses from an LLM to a single request) you need to match the effective utility that is produced by the total number of actual responses that was recorded. However in section 3.2, the paper inverts this and introduces \delta(M) as the "empirical quantity" and doesn't really properly establish the relationship between the two measures. The justification given is that delta between the vote accuracy from independent votes and actual votes is always available to compute but that is also an intuitive metric to understand and perhaps should have been introduced as the primary one.

Another key weakness is that the experiments and results on real LLMs do not actually show any results regarding what the authors intend to establish in the original motivation for the work. The LLM results seem to show that there is no significant difference between vote accuracy from independent votes and actual responses, and also show's that using the metric as a controller saves some budget but doesn't lead to better outcomes, stated in the paper as "moves the stopping point but does not establish better aggregate decisions than raw prefix length." An interesting result could have been if the measurement efficacy but correlation weakness established in the simulations also held in the real experiments.

Questions for authors

What are some explanations for why the experimental conclusions are different between the simulations and real experiments?

What are some plausible mechanisms that govern the how repeated samplings from LLM become correlated or have dependence on each other and why was this not addressed more in the paper?

3 comments
AuthorsOpenReview ↗

Cross-fit correction, the primary measure, hosted results, mechanisms

Thank you for the constructive feedback! Pages refer to the revised PDF.

Correction. The exact-condition cross-fit matched runs by question identifier, but some run families reused identifiers for different questions: 630 of the 752 test trajectories had a reference built entirely from answers to another question. Matching on the question asked (its text and accepted answer) leaves 451 trajectories on 155 questions. Their reference-minus-observed terminal gap is −0.12 percentage points (pp), 95% question-bootstrap interval [−1.13, +0.84], and +0.65 pp [−0.55, +1.94] on the 227 trajectories (95 questions) labelled test in their own run, as in the submitted analysis (Table 5, p. 7). We withdraw the submitted +0.11 pp [−0.75, +1.03] and the positive MATH-500 64-call slice (+9.20 pp, now −2.24 pp [−8.37, +2.39]). Table 32 (App. M, p. 41) lists replaced values.

W1. Which measure is primary. The direct gap Δ(M) of Eq. (2): the plurality utility of M independent draws from the matched answer distribution minus that of the M recorded answers. VOTE-ESS (Eq. 3) is secondary: it restates observed utility as a number of independent calls. With u = U_obs(M), VOTE-ESS equals M exactly when Δ(M) ≤ 0, so the mean ratio to M falls below one whenever some gaps are positive, even at a zero mean gap. 399 of the 451 have VOTE-ESS = M, 51 have 1 and one lies between: the raw ratio is 0.889 at gap −0.12 pp. The abstract, §1 and §3.2 (pp. 1, 3) now lead with the gap; §3.2 states this link. Δ(M) detection by simulated mechanism: L.2 (p. 35).

W2. What the hosted experiments show. For stopping, your reading is right (Table 3, p. 6; Table 6, p. 8). For measurement (all post hoc):

  • Against zero. On all 451, four votes instead of one raise recorded utility by 1.92 pp [1.10, 2.79] and reference utility by 1.76 pp [1.06, 2.51], a difference of −0.16 pp [−0.37, +0.05] (Table 29, p. 38). Two Qwen-Max collections (40 MATH-500 questions) give −0.93 and +0.01 pp (L.8).
  • Power. Four copies of every answer are detected by the terminal gap in only 38% of 1,000 simulations (test-labelled 14%), two copies in 10% (6%); the one-to-four gain gap detects them in 100% and 70.9% (30.6%, 11.2%). The observed gain gap (−0.17 pp) lies inside its independence null (mean −0.22 pp, 95% range [−0.36, −0.09]): no evidence of copying at this power, not evidence of independence. Partial copying is untested (App. M, p. 42).
  • Planted repeats. Copying each of four draws four times raises the 1-to-16 gain gap by 2.50 pp [0.40, 5.11] for Qwen-Max and 2.57 pp [0.72, 5.24] for DeepSeek Flash. Qwen3.7-Plus (91% one-call accuracy) moves by 0.28 [−0.32, 1.06]. This imposes duplication on fitted distributions; it does not measure dependence in calls (L.8, pp. 39–40).
  • Against independence. Under independence, finite reference pools give a negative gap (about −0.5 pp; −1.0 pp test-labelled); the test-labelled gap lies 1.65 pp [+0.22, +3.32] above it, an excess that subsampling the reference does not remove, though only +0.52 pp [−0.99, +2.11] on 166 of them with ≥ 32 reference responses (pp. 34, 42). Shuffling responses across runs gives p = 0.046 and 0.012: they are not exchangeable across runs. The archived trajectories are not re-scored (B.1, p. 17).

Q1. Why simulation and hosted results differ. (1) The simulations plant dependence of known strength (§4, p. 4; §5.2, p. 5); hosted runs have none planted. (2) Hosted answers repeat often, but so do independent draws from a concentrated answer distribution, and the matched reference keeps that concentration.

Q2. Mechanisms. Candidates (closest simulated mechanism; Table 4, p. 7): a cache or seed handling returning earlier answers (exact block replay, Markov copy/redraw); batches sharing serving state (shared batch mixture); a model or configuration change between runs (Dirichlet–multinomial). The non-exchangeability in W2 fits the last candidate or dependence within runs; our records cannot separate the two. An archived Qwen-Max rerun re-sent 65 MATH-500 questions × 16 seeds 1.8 h later: same-seed pairs were identical in 16.0% of 1,040, +9.9 pp over different seeds (clustered 95% [7.6, 12.3]), and agreed in answer +4.3 pp more often [2.3, 6.5], a weak channel our analyses avoid (trajectory calls and cross-fit references use distinct seeds). A pre-registered same-endpoint probe (62 mid-accuracy questions) gave identical text in 2/62 pairs each for a new seed sent twice serially or concurrently and in 2/62 and 1/62 without a seed, against 0.65% for different seeds; upper 95% bounds give ≤ 10.4 pp excess identity and ≤ 0.33 answer copying (post hoc), below the frozen r* = 0.35: seed reuse and identical requests are at most weak channels here. No response ID repeated; no cached token was reported (implicit caching needs ≥ 1,024 prompt tokens; ours ≤ 210); concurrency was client-side (server batching unobserved); we cannot yet attribute a cause (App. M, pp. 42–43).

Thanks for addressing W1 on which measure is primary and for clarifying/updating that. For W2, which is critical, I don't think value of/impact from measurement is a significant enough contribution, and the updated experiments/correction do not provide, additional clarification or insight into when additional calls stop helping or providing new information on real LLM invocations.

AuthorsOpenReview ↗

Where additional calls changed voting accuracy

Thank you for clarifying the remaining concern. We revisited the actual Qwen-Max calls to distinguish two questions: how voting accuracy changes with the number of calls, and whether those gains fall below an independent reference. A small terminal reference gap alone cannot identify a useful stopping point.

Here are the voting curves on three recorded collections. Within each row, every budget uses the same questions and the corresponding prefix of their calls. The two later collections use the same 128 questions with new calls. We use the formatting-equivalence scorer of Appendix B.1, with fractional credit for tied plurality decisions.

CollectionQuestions4 calls8163264
Original holdout4070.00%73.75%73.75%75.00%72.50%
Later 32-call collection12870.51%70.70%72.81%73.83%—
Later 64-call collection12872.46%73.57%72.66%75.00%75.00%

Appendix L.3 describes these collections, including “Two later 128-question runs.” We additionally computed the paired changes between adjacent budgets from the saved per-question outcomes. In the later 64-call collection, going from 16 to 32 calls improved scored correctness on four questions and reduced it on none: +2.34 percentage points, with a pointwise 95% question-bootstrap interval of [+0.39, +5.08]. Going from 32 to 64 changed no question’s plurality score. These are post-hoc descriptive comparisons; the interval is not adjusted for examining multiple budgets, and the observed plateau does not establish a zero effect in future calls. Under the archived scorer, 32→64 gives +0.39 points from one fractional-tie outcome; both formatting-equivalence variants give zero.

The original holdout also shows why a short plateau is insufficient for stopping: scored correctness was unchanged on every question between eight and sixteen calls, yet one question improved between sixteen and thirty-two. In the later 64-call collection, both adaptive rules stopped after about 3.6 calls on average and reached 69.53%, compared with 75.00% at fixed 32. That accuracy difference alone does not favor the larger budget: the stopping objective penalizes each call by 0.01. The relevant comparison is between the learned and raw-prefix rules, which made essentially the same decisions; the learned measure did not improve that cost–accuracy tradeoff. Describing the full voting curve remains distinct from predicting a useful cutoff from the available prefix.

The separate reference comparison asks whether repeated calls achieve lower voting accuracy than independent draws from a question-matched reference distribution at the same budget. Across all 451 matched trajectories, the corrected terminal gap is −0.12 points [−1.13, +0.84], so that aggregate contrast does not establish an overall deficit. The concrete finding from the budget curves is that gains can resume after an intermediate plateau, while doubling the largest recorded budget can leave voting accuracy unchanged. Neither a near-zero terminal gap nor an intermediate plateau is sufficient evidence that no useful calls remain. This concerns scored answers; unchanged voting accuracy does not establish that the response text contains no additional information.

We appreciate your asking for this connection to actual invocation budgets. These comparisons make the hosted finding more concrete: they show where further calls changed voting accuracy, why an intermediate plateau can be misleading, and why the reference gap alone does not justify stopping.

Comments on the paper

2 comments

Dear Reviewers,

The authors have replied to your initial comments. Please have a look and engage in the discussion.

Thanks!

Your AC

AuthorsOpenReview ↗

What the corrected comparison tells us about repeated calls

We appreciate the close reading of how we matched repeated calls across runs. In the submitted Qwen analysis, different runs sometimes used the same identifier for different questions. We corrected the comparison by matching the question text and accepted answer across runs. Across 451 trajectories on 155 questions, the vote accuracy expected from independent draws minus the observed vote accuracy was −0.12 percentage points at the final call budget (95% interval −1.13 to +0.84). The revised analysis withdraws our earlier claim of a positive gap on the 64-call MATH-500 subset (Table 5, p. 7 and Appendix M, p. 41).

A small gap at the final budget can hide substantial dependence. In a simulation that copied every answer four times, the final-budget comparison detected the copying in 38% of datasets. Comparing the gain in voting accuracy from one call to four with the gain expected under independent draws detected it in all of them. The observed Qwen result was within the range expected under independence (Appendix M, p. 42). For choosing when to stop, the learned rule saved a fraction of a call per question on both Qwen and DeepSeek. Neither provider showed a clear improvement in the combined accuracy and cost measure over simply using the number of calls so far (Table 6, p. 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.