- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes grey-box feedback that uses the full probability vector over candidate answers. It formulates adaptive inference as sequential mode identification with distribution-valued observations. The authors characterize the optimal asymptotic stopping rate in this setting. They propose ASC-D, which bets on probability differences between candidate answers and stops sampling when sufficient evidence has accumulated. With an appropriately designed betting grid, the algorithm attains the theoretically optimal characteristic time. The experiments include synthetic distributions, MMLU-Redux, and additional ARC-Challenge results in the appendix. In empirical replay experiments on a shared set of 90 MMLU-Redux questions, ASC-D uses 46.4%–95.6% fewer trajectory observations than PPR-1v1.
Critical strengths
- The paper clearly characterizes the informational value of full probability distributions relative to single answer labels.
- The stopping rule is closely aligned with the theoretical analysis.
- The experiments help distinguish the contributions of feedback information and the stopping rule.
Critical weaknesses
- The real-model experiments mainly validate replay performance on empirical distributions, with limited guarantees for the original model distribution.
- The conditions for termination guarantees with a finite betting grid need further clarification.
- The main results rely on a filtered shared question set, whose representativeness needs further validation.
Questions for authors
- Do the current certification results concern the empirical population or the original model’s answer distribution? Please clarify the statistical relationship between the empirical population and the original model distribution. In particular, how uncertain is the reference mode estimate when the modal gap is small? Are there experiments that determine the reference mode using an independent trajectory pool? Such experiments would then use newly generated trajectories to evaluate stopping times and incorrect-certification probabilities. This information will affect my assessment of online sampling savings and certification validity.
- How does the practical betting grid ensure termination, and how does this relate to the δ-PAC definition? Please provide the betting grid used in each experiment and explain how it was chosen. Is eventual termination guaranteed at a fixed confidence level when the distribution is unknown and the modal gap is extremely small? If not, how are runs handled when the budget is exhausted? How are the probabilities of incorrect certification and non-certification measured separately? This information will affect my assessment of the theoretical guarantees and the conditions under which the algorithm applies.
- Do the main conclusions still hold on the full set of 120 questions? Please report stopping times, correct-certification rates within the budget, and the proportion of uncertified runs on the full question set. Please also explain which selection criteria the 30 excluded questions failed to meet. If some runs reach the budget limit, how are they handled when computing the average number of trajectories? Please clarify how reusing this question set affects comparisons across models. This information will affect my assessment of the reported savings and the representativeness of the results.
We thank the reviewer for raising these questions and provide our clarifications and additional results below.
W1/Q1: Empirical populations and reference-mode uncertainty. The certification results in Exps. 1–2 concern the empirical population F=M−1∑i=1Mδθi, where M is the number of stored probability vectors. Its mode is computed exactly from πk=M−1∑i=1Mθi,k, so no additional reference pool is needed for evaluation under F. We conduct hundreds of paired replays per question and setting to reduce Monte Carlo error in the performance estimates under F.
The relationship to the original model distribution F involves sampling uncertainty. Under i.i.d. sampling from F, Hoeffding's inequality gives maxk∣πk−pk∣≤εM w.p. at least 1−α, where εM=log(2K/α)/(2M). On this event, the empirical and original modes agree whenever the empirical gap exceeds 2εM. Small gaps can leave this agreement unresolved. This does not affect evaluation against the exact empirical mode.
In addition, we added Exp. 3/Table 5 to measure fresh-sampling costs (trajectories, tokens, and runtime). We have not established original-distribution certification-error rates using an independently validated reference mode.
W2/Q2: Practical grids and budgeted certification.
(1) Grids. ASC-D uses Λ2,8 (15 points) in Table 2 and the Dirichlet-concentration study, Λ1.5,16 (32 points) in the characteristic-time study (Appendix D), and 0,0.01,…,0.99 (100 points) in all real-model experiments. Geometric grids densify both endpoints. The uniform grid fixes numerical resolution and is shared with matched controls.
(2) Termination. The δ-PAC definition requires almost-sure termination and error at most δ. A fixed finite grid need not ensure termination for arbitrarily small gaps. Our optional Hoeffding backup supplies this guarantee and preserves the applicable asymptotic rates. See Reviewer NaQQ's W2/Q1 and Remark 3.1 and Appendix B, Corollary B.1.
(3) Budgeted outcomes. In fixed-budget certification comparisons, non-crossing runs are uncertified and charged N; Num. Gen. is the mean min(τ,N). Relative to the exact empirical mode a⋆, we estimate
p_{\mathrm{nc}}(N)=\mathbb{P}(\tau > N)$$ from fractions of all replays. Correct certification, incorrect certification, and non-certification sum to one, in which $\delta$ controls incorrect certification. W3/Q3 reports full-set results. The accuracy-matched ASC comparison permits a budget-end estimate, which is not a certificate. True-mode error evaluation with fresh trajectories is discussed in W1/Q1. **W3/Q3: Question selection and full-set evaluation.** We first clarify the selection used in *Exp. 1*. Of the original 120 questions, 17 have different ordered top-two answers across $T_a\in\\{4,8,16,32\\}$ in the Qwen3-1.7B analysis. Among the remaining 103, another 13 fail to reach 95% correct recovery for at least one method at a tested temperature within 32,768 trajectories. The retained 90 keep the leading answer and closest competitor fixed while varying the gap, allowing required-budget estimates within the cap. We repeated *Exp. 2* on all 120 questions for all three models at $T_a=1$ and $N=128$. Every run contributes $\min(\tau,N)$ to mean Num. Gen.; uncertified runs are charged $N$. Corr. Cert. and Uncert. are percentages of all runs that correctly certify the empirical mode by $N$ and that do not stop by $N$, respectively. The table gives $95\\%$ results. ASC-D uses fewer trajectories and achieves higher correct-certification rates than both baselines on every model, also at $97.5\\%$ and $99\\%$ confidence. Full results appear in *Appendix E.4, Table 13*. | Model | Method | Num. Gen. | Corr. Cert. | Uncert. | |---|---|---:|---:|---:| | Llama-3.2-1B | ASC-D | **79.75** | **57.21%** | **42.77%** | | Llama-3.2-1B | CITE-4 | 98.48 | 35.87% | 64.10% | | Llama-3.2-1B | PPR-1v1 | 100.05 | 34.41% | 65.58% | | Llama-3.1-8B | ASC-D | **48.62** | **78.67%** | **21.32%** | | Llama-3.1-8B | CITE-4 | 61.46 | 68.52% | 31.46% | | Llama-3.1-8B | PPR-1v1 | 63.40 | 67.22% | 32.77% | | Qwen3-1.7B | ASC-D | **36.68** | **83.49%** | **16.49%** | | Qwen3-1.7B | CITE-4 | 39.88 | 81.64% | 18.35% | | Qwen3-1.7B | PPR-1v1 | 41.56 | 81.00% | 18.98% | *Exp. 1*'s filtering enables controlled gap scaling; *Exp. 2* does not require it. We reused the Qwen-selected 90 questions for consistency with *Exp. 1* and common composition across models. This selection can influence cross-model averages, whereas the unfiltered 120-question evaluation removes that filter.