- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to select the best of K fixed AI agents under a limited evaluation budget, and proposes DASE. In each round, DASE picks a leader and a challenger and generates candidate queries from the evaluation history. It then evaluates all agents on the query with the largest estimated gap plus an uncertainty bonus. The paper gives bounds on diagnostic-query acquisition and correct selection, and runs experiments on a synthetic task, MATH, and clinical decision-making.
Critical strengths
- The problem is practical. Agents often differ only on a few rare queries, and the paper formalizes this with pairwise diagnostic regions.
- The framework is modular. The theory handles adaptively chosen queries without assuming independence, and the authors state openly that their bounds do not show an advantage over random sampling.
- The experiments cover synthetic, math, and real clinical data. The three pool settings (Standard, Small, Restricted) show when generation helps. At larger budgets DASE beats Random by a wide margin (0.484 vs. 0.369 on the clinical task at T=150).
Critical weaknesses
- Query generation is never tested on real data. In MATH and the clinical study, “generation” means retrieving items with recorded outcomes from an extra pool, so what it mainly adds is access to more data. Its gains show up mostly in the Restricted Pool, which leaves out the diagnostic region by design. The LLM variant only matches Random (0.340).
- The baselines are weak, and the setups favor the method. The paper compares only against Random and a variance-based QBC. Stratified sampling with Neyman allocation, active testing [1], and active model selection for LLMs [2], which the paper itself cites, are all missing. In each study the key difference lines up with one hand-picked feature: Number Theory for the constructed NT-Diagnostic agent, and the pathology label in the clinical features. A simple stratified baseline might recover most of the gain.
- The clinical results do not support high-confidence selection. The top two agents differ by 0.0176 on the 500-case reference set, with a paired SE of about 0.022, so which agent is truly best is itself uncertain. The paper also reports no full-information ceiling. Selection accuracy stays below 0.5, and DASE gives no confidence statement for its choice.
References
[1] Active Testing: Sample-Efficient Model Evaluation. [2] Active Model Selection for Large Language Models.
Questions for authors
- If a stratified sampler with Neyman allocation used the same features (subject in MATH, pathology in the clinical study), how much of DASE’s gain would remain? Do the results hold without the pathology label and without the constructed NT-Diagnostic agent?
- In the clinical study, what accuracy does an oracle reach if it evaluates every admission available for querying? Can the best agent be statistically distinguished from the runner-up on the reference set?
- Could you run a small experiment with real generation, such as programmatically generated math problems with checkable answers, and count generation and validation costs in the budget?
Stratified baselines, oracle ceilings, and real generation
We thank Reviewer oud4 for the concrete experimental questions. We ran all the proposed experiments using the paper's evaluation protocols. The new results appear in blue in the revised manuscript.
Response to Comment oud4-1: Would Stratified+Neyman recover the gain? Do the results hold without the pathology label and without the constructed NT-Diagnostic agent?
Stratified+Neyman. We implemented this baseline using subject metadata on MATH and pathology metadata in the clinical study. Allocations follow nh∝Nhσh, with σh re-estimated from the evaluation history each round. This baseline now appears alongside the other methods in the blue rows of Tables 1-2.
Within-stratum variance is dominated by the many queries on which weak agents disagree. As a result, Neyman allocation does not prioritize the stratum containing the region that separates the leader and challenger. Using the relevant category for stratification therefore does not provide the same benefit as acquisition focused on this pair.
Without the pathology label. In the clinical benchmark, nearly all grouping signal is concentrated in the pathology variable. Removing it leaves the methods with low selection accuracy, typically around 0.3-0.4 at T=150. After removing the pathology indicators, DASE Pool Only achieves 0.505 selection accuracy, exceeding all baselines using this representation. On MATH, we tested both removing the subject indicators and replacing them with text features (revised Appendix D.5). DASE Pool Only reaches 0.928 with the subject indicators removed and 0.878 with text features, compared with 0.860 and 0.850 for Random at T=200. MATH remains informative under these ablations because other feature dimensions retain signal associated with the subject categories. In particular, diagnostic problems concentrate at high difficulty: 27% of Level-5 problems are diagnostic, 3.5 times the Level-1 rate. Difficulty therefore helps locate the diagnostic region even without subject indicators, and DASE Full reaches 0.970 with difficulty as its only feature.
Without the NT-Diagnostic agent (K=4). Selection becomes much easier. The top-2 gap widens from 0.010 to 0.061, the two leading agents differ on over 1,200 problems rather than 168, and every method reaches at least 0.93 at T=200. Random, DASE Pool, and Stratified+Neyman each reach 0.996, while TradQBC reaches 1.000. NT-Diagnostic is designed to create the sparse diagnostic region studied in the paper. When the leading agents differ across many problems, as in this ablation, uniform sampling suffices.
Response to Comment oud4-2: Oracle ceiling and whether the best agent is statistically distinguishable
We computed both quantities using the same clinical evaluation protocol:
Oracle ceiling. An oracle that evaluates all 200 admissions available for querying in a trial reaches 0.491 with the experiments' outcome noise and 0.515 with noiseless lookups. DASE Full reaches 0.484 at T=150, nearly matching the full-query oracle with three quarters of its budget. The oracle result shows that accuracies below 0.5 also occur when every available query is evaluated, reflecting the difficulty of this problem instance. The revised text reports these oracle ceilings alongside the clinical results.
Distinguishability. We now report the calculation raised by the reviewer. On the 500-case reference set, the top two agents differ by 0.018 with a paired SE of 0.022 (z=0.82), differing on 119 of the 500 cases. The target k∗ is defined by performance on this fixed reference set, so correct selection remains well-defined even when the gap is small relative to noise from sampling subsets. Every method faces the same target under the same noise. The comparison, DASE 0.484 vs. Random 0.369, therefore measures the benefit of acquisition under a common evaluation protocol.
Response to Comment oud4-3: A small experiment with real generation, with generation and validation costs in the budget
We added a real-generation experiment on MATH (revised Appendix D.7). A programmatic generator uses the acquisition's anchors to create problems with automatically verifiable integer answers. We calibrate each template family for difficulty by matching the self-consistency strategy's accuracy on real MATH problems at each level. The GPT-4o-mini strategy agents answer each selected problem live, with each evaluation counting as one unit of the budget T. We evaluate the calibrated generator over 50 trials, with all baselines replayed on the same seeds. The acquisition steers generation toward the diagnostic region (62% of generated problems are Number Theory), and the calibrated generator matches retrieval, with a +0.200 ± 0.086 paired advantage over Random.