AURA: Measuring Evidence from Repeated Calls to Hosted Foundation Models

Repeated sampling is a standard way to improve foundation-model answers, but on a hosted service each sample is a billed API call. We introduce AURA, an output-only framework for asking how many independent categorical votes would match the plurality utility of a repeated-call trajectory. The resulting VOTE-ESS is an operational matched-budget index, not an estimator of latent correlation. We validate it on 180 Dirichlet–multinomial configurations and an exact block-replication control: it decreases with dependence in every fixed configuration and recovers the block budget exactly in 96.3% of cases. In a broader 288-cell stress test spanning four dependence mechanisms, an interpolated isotonic categorical VOTE-ESS reduces non-i.i.d measurement error by 86.4% relative to the strongest output-only proxy. Its exact-block compatible intervals, however, cover only 83.5%, below a prespecified 90% coverage threshold, and its stopping controller incurs 0.0324 more regret than a raw-prefix controller. On frozen Qwen trajectories, the categorical raw/isotonic terminal ratios are 0.944/0.981 over 2,052 disjoint test trajectories, while the direct reference-minus-observed plurality gap is $-0.0063$ (95% question-bootstrap CI $[-0.0101,-0.0024]$). An exact-condition cross-fit over 752 test trajectories instead gives $+0.0011$ $[-0.0075,+0.0103]$. Breadth checks over 69 aliases and full matched runs on two official DeepSeek models show that the protocol transfers, but not that all providers are independent. Finally, prefix-only controllers save about 0.22–0.25 calls without detectable gains in accuracy or matched objective. These results support a disciplined way to measure repeated-call evidence while showing that measurable shrinkage need not justify a different test-time compute policy.