- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies cost-optimal evaluation for certifying updates to LLM serving configurations. It formulates the problem as fixed-confidence decision identification with heterogeneous evaluation costs, introduces an instance-dependent characteristic certification cost, and proposes the method `Track-and-Certify'. The paper establishes an information-theoretic lower bound and proves first-order asymptotic optimality under finite, stationary, identifiable observation models. Experiments with frozen LLM model pools demonstrate substantial savings from adaptive evaluation allocation.
Critical strengths
-
The problem of minimizing pre-deployment evaluation costs is interesting and practically relevant and distinct from optimizing inference-time routing.
-
The characteristic certification cost provides a useful interpretation of when richer same-query feedback is economically beneficial.
Critical weaknesses
Please state the theory contribution beyond prior controlled sensing results. For a nonempty alternative set with a finite characteristic cost, the cost program can be written as
=\max_{v\in\Delta(\mathcal Z)} \min_{\lambda\in\operatorname{Alt}(\theta)} \sum_z\frac{v_z}{h_z}d_z(\theta,\lambda),$$ where $v$ is the share of cost assigned to each action. A closely related reference missing from the manuscript is Nitinawarat and Veeravalli (2015) [R1]. It treats non-uniform control costs and gives an asymptotically optimal expected-cost test. Its memoryless case gives the same information-per-cost form when all other hypotheses must be ruled out. This is a separate work from Nitinawarat et al. (2013), which the manuscript already cites. Composite controlled sensing also excludes parameters in the same correct hypothesis; see Deshmukh et al., cited by the paper, and Lemma 1 and Theorems 1–3 in its [public preprint](https://arxiv.org/abs/1910.12697v1). Another relevant reference not listed in the manuscript is Degenne and Koolen (2019) [R2]. Section 2 defines the alternatives to an answer as the models where that answer is incorrect, which is the same form as Equation (7). Their treatment of multiple correct answers also gives useful context for the role of your unique-true-optimum assumption. The models and answer assumptions in these references differ from yours, so the comparison should state both the shared ideas and the new results. The paper already credits controlled sensing and Track-and-Stop. Please make the remaining new work precise: the decision-changing alternative set, ties and multiple correct configurations, the finite-model certifier, and the LLM observation design. Cost weighting and the tracking-and-stopping structure alone do not establish a new general theory. For the LLM related-work discussion, like RACER (Hao et al., 2026) [R3], relevant works are missing from the reference list. For example, RACER calibrates model sets with abstention and controls their expected miss risk under exchangeability; its guarantee averages over the calibration data and a new query. These works optimize model-set size or annotation cost, while this paper optimizes the evaluation cost of identifying a serving configuration. A short comparison in Section 2 would help clarify these different cost objectives and risk guarantees. [Reference] - **[R1]** S. Nitinawarat and V. V. Veeravalli. “Controlled Sensing for Sequential Multihypothesis Testing with Controlled Markovian Observations and Non-Uniform Control Cost.” *Sequential Analysis*, **34**(1):1–24, 2015. [DOI: 10.1080/07474946.2014.961864](https://doi.org/10.1080/07474946.2014.961864). [Open preprint](https://arxiv.org/abs/1310.1844v2). - **[R2]** Rémy Degenne and Wouter M. Koolen. “Pure Exploration with Multiple Correct Answers.” *Advances in Neural Information Processing Systems*, **32**, 2019. [Official proceedings and paper](https://proceedings.neurips.cc/paper_files/paper/2019/hash/60cb558c40e4f18479664069d9642d5a-Abstract.html). - **[R3]** S. Hao, H. Zeng, H. Wei, and B. Jing. “RACER: Risk-Aware Calibrated Efficient Routing for Large Language Models.” ICML 2026.Questions for authors
See Weaknesses. And:
-
The proposed framework assumes a pre-specified quality–cost trade-off parameter , but provides limited guidance on how this parameter should be selected in practical LLM serving systems. This is an important omission because directly determines the optimal serving configuration, the set of decision-changing alternatives, and consequently the characteristic certification cost. So, I have three questions:
- How should practitioners choose based on operational budgets or quality requirements?
- How sensitive are the optimal serving decisions and certification costs to changes in , particularly near decision boundaries?
- Can the framework be extended to certify optimal serving configurations uniformly over a range of values, rather than requiring a single pre-specified trade-off parameter?
A sensitivity analysis over , including the corresponding changes in optimal configurations and certification costs, would substantially improve the practical relevance of the proposed framework.
-
And in Appendix C.2, the stated adaptive Hellinger bound appears to have an unconditional expectation on the left but a random action count on the right. Could the authors provide a rigorous deterministic bound that accounts for adaptive sampling, and clarify how it yields the claimed tail bound for the MLE stabilization time?
8 comments
Response to Reviewer SRnX — Part 2/4
The Hellinger step in Appendix C.2 (Question 2)
Thank you for catching this. The reviewer is right: as written, the left-hand side is a deterministic expectation while is random, so the displayed inequality is not well-posed under adaptive sampling. The conclusion (Eq. 20 in the revision, ) is unchanged. We have replaced the step with the following argument, now in revised Appendix C.2.
Martingale. For a false , let . This is positive by uniform common support and equals 1 iff . Define Since is -measurable and , we have . Hence is a nonnegative martingale with .
Deterministic bound. By global identifiability, choose with . All affinities are at most 1, and the forced-exploration rule gives the deterministic coverage (Eq. 18). So on every sample path, . Therefore which is deterministic.
Tail of the stabilization time. Let . If , then for some and some ; this covers any tie-breaking rule. By Markov's inequality and a union bound over the finitely many and over , with , for : Appendix C.3 uses only Eq. 20 and is otherwise unchanged. While revising C.2–C.3 we also tightened four details:
- We prove the coverage bound (Eq. 18) explicitly. It holds deterministically with and , and the number of forced rounds satisfies , whichever underfilled action is chosen.
- We state that and every are read off a single never-stopped process.
- We cover small with the constant .
- We add the one-line argument that .
Response to Reviewer SRnX — Part 3/4
The trade-off parameter (Question 1)
(1.1) How to choose . The utility prices serving cost in units of correct answers, so is the value of one additional correct answer. There are three practical routes, now described in a new Appendix D.10:
- Value-based: set if a correct response is worth dollars.
- Budget-based: treat as the Lagrange multiplier of a per-query serving budget. Utilities are linear in , so the optimal configuration is piecewise constant in ; take the smallest whose optimum meets the budget.
- Quality-based: take the largest whose optimum meets a minimum accuracy.
The budget-based and quality-based routes recover only the configurations on the upper concave hull (Pareto frontier) of (cost, accuracy) points. Because they depend on unknown accuracies, should be fixed from independent historical data, as for the evaluation family in Cor. 1. We note both points explicitly.
(1.2) Sensitivity. We varied over a threefold range (, ) for all 11 pools, holding , the evaluation costs, and fixed (new App. D.10, Fig. D.9).
- Every utility is linear in , so optimal sets and are piecewise constant. There are 89 breakpoints, giving 100 constant intervals; we recomputed all 6,010 characteristic programs independently.
- Decisions. The true optimal configuration changes in only one pool (M2, at ).
- Family rule. moves with even when the decision does not. The family rule is unchanged in 10 of 11 pools. For five of these the result is trivial, because or both costs vanish. M10 crosses the threshold.
- Replay at shifted (4 pools, 11 operating points, 2 families × 100 seeds; new Table D.8).
- With Algorithm 1's stopping statistic, no run reached the horizon and 2 of 2,200 were wrong, consistent with .
- Realized cost is 0.98–1.26× the first-order prediction unless is tiny.
- Moving can also make certification cheaper, because it can remove decision-changing environments from a fixed finite class.
- Table D.8 also reports our original replay statistic, which tests against all other environments. It coincides with Algorithm 1's on every main experiment (App. D.1) but is overly conservative at shifted .
(1.3) Certifying uniformly over . Yes, under one additional assumption: under each environment, no two configurations have identical utility lines in .
- The answer. Utilities are linear in , so each environment induces a piecewise-constant map with finitely many breakpoints. Take this map, up to its breakpoints, as the object to certify. It is unique under the assumption, and since is finite there are finitely many such answers.
- The alternative set. It is the set of environments whose optimal configuration differs from the truth's on a subset of of positive length. Using this set, rather than a union of the fixed- sets, avoids the truth's own breakpoints, where those sets depend on the tie-breaking selector.
- The theory. Theorem 1, Algorithm 1, and Theorem 2 then carry over, because their proofs use only a finite answer set and the alternative-set definition.
- The cost. , with the supremum over that are not the truth's breakpoints. So uniform certification is at least as costly as certifying the hardest single .
- When the cost is finite. It is finite only because is finite. If the interval contains a true breakpoint, a richer class would make it diverge (Eq. 11), so a practical variant excludes -neighbourhoods of breakpoints.
- Fixed- repetition. Repeating fixed- certification at several values does not give this simultaneous guarantee.
This is stated as a remark in App. D.10 and is not evaluated empirically.
Response to Reviewer SRnX — Part 1/4
Theory contribution relative to controlled sensing and pure exploration
We thank the reviewer for the expert and constructive review and for the precise pointers. We agree with the main point. Viewed as general results, our lower bound and Track-and-Certify instantiate known machinery going back to Chernoff (1959), and the submission did not say so clearly enough. The revision adds a short paragraph to Sec. 2 ("Relation to controlled sensing") and a detailed mapping in App. B.1 with a comparison table (Table B.2). It also adds the cited references (Nitinawarat & Veeravalli 2015, Degenne & Koolen 2019, Chernoff 1959).
Shared or adapted (now stated explicitly).
What we claim as new.
Ties and multiple correct configurations (new App. C.4, referenced right after Theorem 2).
We hope this makes the scope precise, and we are glad to adjust the wording if any item still reads as overstated.