- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work presents a Bayesian sequential evaluation method to address uncertainty caused by stochastic LLM-system behavior when the same benchmark prompt is evaluated repeatedly. It estimates each prompt's success probability with an independent Beta-Bernoulli posterior, propagates those posteriors to W_mean, W_min, and W>ν, and formulates prompt selection as a bandit problem whose reward is expected reduction in the variance of W>ν. Experiments show that a single greedy run can overstate reliable behavior; greedy and Thompson sampling generally reduce posterior variance and MAE faster than round-robin; and JailbreakBench at the 0.95 threshold is an exception because most harmful prompts are almost always refused.
Critical strengths
-
The paper makes within-prompt stochasticity directly useful for evaluation decisions. Instead of only attaching an interval to a benchmark mean, it estimates a posterior for each prompt and propagates it to W_mean, W_min, and W>ν. This supports reliability questions that a single mean cannot answer, such as whether a behavior is consistently above a threshold or whether the worst prompt rarely produces the desired behavior.
-
The adaptive design is supported by useful evidence and honest negative results. Across four behavior domains, greedy and Thompson sampling generally reduce posterior variance and MAE faster than round-robin; JailbreakBench at the 0.95 threshold is correctly identified as a case with little benefit. Threshold/prior sweeps, coverage checks, a two-system comparison, ordinal extensions, and released generations make the practical scope substantially easier to assess.
Critical weaknesses
-
The main experiments do not test whether the credible intervals are actually reliable. The sequential results show that adaptive sampling approaches the posterior computed from all stored generations, but that full-data posterior is not the true value of W. For the four main binary case studies, the paper does not report coverage or another ground-truth/held-out check showing that intervals for W_mean, W_min, or W>ν contain the true aggregate at their nominal rates. A synthetic calibration study or a larger held-out validation pool would make the uncertainty claims much stronger.
-
The efficiency claim is demonstrated in a narrow replay setting. Each benchmark evaluates essentially one system or pair. The agentic case studies use only 15 and 27 prompts, and the sequential experiments replay a fixed pool of 100 precomputed generations per prompt. The main comparison is with round-robin and uses the full stored-generation posterior as the primary MAE reference. Appendix C.3’s alternative references and Appendix D’s threshold sweeps are useful, but a smaller online study with additional systems and stronger adaptive or uncertainty-aware baselines would better establish practical robustness.
Questions for authors
-
How should the threshold ν and aggregate W be chosen before observing the data? The concern is that Section 3 derives the sequential reward specifically for W>ν, while Appendix D evaluates several thresholds; post-hoc inspection may require multiplicity control.
-
Can the main efficiency claim be tested in a smaller online study with additional systems and stronger baselines such as bootstrap, hierarchical Bayesian, or stratified allocation? This would address the replay setting in Appendix C.1, the full-data posterior reference in Appendix C.3, and the current one-system-per-domain scope more directly.
3 comments
-
Stratified allocation saves budget on one of the five pairs (ALFWorld-Qwen, 17×M vs 20×M). Neyman allocation is designed to minimize the variance of the mean, not of a threshold count, which is why our reward is defined on the aggregate.
-
The bootstrap does not fit this problem. A prompt whose generations so far all agree has zero bootstrap uncertainty, so it is never sampled again, and its interval has zero width. Bootstrap intervals are also unreliable for a minimum () and are not justified when sample sizes depend on the data. Its MAE at 10×M is 2.1–3.1 times greedy's (MT-Bench, ALFWorld, GAIA). We may be misreading the suggestion. We would be glad to evaluate a specific bootstrap procedure if the reviewer has one in mind.
-
We read "hierarchical Bayesian" as partial pooling across prompts, i.e., relaxing our independence assumption, which we list in Section 5 as future work. A hierarchical prior extends our method rather than competing with it as a baseline. It changes the prior, not the method: our allocation rule and Monte Carlo aggregates apply under a hierarchical prior, with the reward computed by Monte Carlo. As a first check, allocating with a shared empirical-Bayes prior was less accurate than independent priors at 10×M on all six pairs (GAIA 1.89 vs 1.14, ALFWorld 0.96 vs 0.71). If the reviewer had a different model in mind, we would be glad to add it.
-
APT (Locatelli et al., 2016), which also targets , is within 3×M of greedy on every pair (Appendix C.5; See our response to Reviewer i4CJ for more on this).
Q1. How should the threshold ν and aggregate W be chosen before observing the data?
We agree that and should be fixed before data collection. They were fixed for all our experiments, and we have now stated this explicitly in Section 3.
Choosing and : These are not tuning parameters; they define the evaluator’s specific question. For example, a safety team wanting to ensure harmful prompts are refused 95% of the time uses with . Appendix D is strictly a sensitivity analysis, and not a selection step. We report all 25 benchmark and threshold configurations without cherry-picking. In fact, our main-paper values are not the most favorable: as reported in Section 4.3, on JailbreakBench yields no sequential gain at all, while selecting for JBB would result in 54% budget reduction (Appendix D). In a new JBB study (Appendix B.2.1) with Qwen2.5-7B-Instruct, which refuses far less often, greedy saves more than 68% of the round-robin budget at the same .
Multiplicity: Because the allocation rule depends only on the outcomes observed so far, the design is ignorable. The target only dictates sampling efficiency, not inference validity. The posterior over is the exact same function of the data regardless of the sampling target, meaning every aggregate (, , ) is derived from that single valid posterior. An evaluator with several thresholds in mind can therefore report the joint posterior of , computed from the same posterior draws, as a simultaneous credible band, with no post-hoc correction. Because each is a sum over prompts, a reward summed over several values of would remain a per-prompt score;
We thank Reviewer aDKB for your constructive feedback, which has helped us substantially improve the paper.
As the author–reviewer discussion period ends soon, we wanted to ask whether you have any remaining questions or concerns. We would be happy to provide further clarification.
We hope our responses have addressed your concerns. If so, we would be grateful if you would consider updating your rating.
We sincerely thank Reviewer aDKB for the positive feedback. We are especially glad you highlighted that our framework makes within-prompt stochasticity directly useful for evaluation decisions, and that our inclusion of various experiments makes the practical scope substantially easier to assess.
W1. The main experiments do not test whether the credible intervals are actually reliable
We now report coverage against known truths (Appendix C.5). The figures below are the share of runs in which the 90% credible interval contains the true value, over 1,000 runs.
With θm drawn from the prior, the intervals cover the truth in 91–97% of runs for W>ν, 87–92% for Wmean and 88–92% for Wmin. These ranges hold for greedy, Thompson sampling and round-robin, at every budget, and for the (M,ν) of all four benchmarks. This is expected: our allocation depends only on outcomes already observed, so it does not enter the likelihood and the posterior is unaffected by it.
The 90% intervals for W>ν and Wmin stay at or close to nominal coverage on truths matched to each benchmark. With θ∗_m drawn as in our response to i4CJ, coverage of W>ν (%, range over the three algorithms) is:
MT-Bench is the only case below nominal (85–89% at 5×M, 91–94% by 20×M). The Wmin intervals cover the truth in 88–93% of runs on MT-Bench, ALFWorld, GAIA, and ALFWorld with Qwen2.5-7B at every budget from 2×M. The exception is JBB with gpt-4o-mini at ν=0.95, where 94 of 100 prompts are refused in every stored generation. There, the W>ν intervals cover the truth in no run up to 20×M (they under-count the prompts that clear 0.95).
Together, these checks show that the W>ν intervals cover the truth at the nominal rate under adaptive allocation, against known truths and against held-out generations.
W2.The efficiency claim is demonstrated in a narrow replay setting
Q2. Can the main efficiency claim be tested in a smaller online study with additional systems and stronger baselines
Please see our response to Reviewer i4CJ. In a new experiment with fresh draws from a known truth, with no stored pool and no Wpost, greedy saves 40–74% of the round-robin budget on five benchmark–model pairs, in line with the replay results (Appendix C.5, Table 3).
On an online study. In the known-truth experiment, each pull is a fresh draw from the prompt's true success probability, which is what a live query to the system returns. The true probabilities are drawn from the posterior given each prompt's stored generations, so the truth is known and realistic. This reproduces the conditions of an online study (fresh draws and an unlimited pool) while keeping the true answer known, on six benchmark–model pairs (the five in the table and JailbreakBench with gpt-4o-mini). Our ordinal LLM-as-judge study (Appendix G) already uses this design with the system's own probabilities, where we show that adaptive allocation saves 81–88% of the budget with 98–100% coverage (Dirichlet model).
On additional systems. The revision evaluates four systems on JailbreakBench (gpt-4o-mini, gpt-5.4-nano, Qwen2.5-7B, Qwen3.5-4B; Appendix B.2.1), adds a second model on all 140 ALFWorld
valid_seentasks (Appendix B.3.1), and extends GAIA from 27 questions to all 53 Level 1 questions.On stronger baselines. We thank the reviewer for these suggestions. We ran all three suggested comparisons in the known-truth setting (Appendix C.5, Table 4). All rules are scored with the same posterior, so only the allocation differs. Budget to match round-robin's accuracy at the full budget: