(-) I found the paper insufficiently positioned within ensemble selection and pool-management research. Section 2 mentions stacking, aggregation, diversity, and safe screening very superficially, but does not sufficiently connect the proposed problem to established selection from model libraries [1], local classifier selection & weighting [2,9], or adaptive pool maintenance [3]. Although these approaches may not tackle exactly the same screening problem, they are important for explaining what is new and which practical ensemble-maintenance decisions the method addresses. Moreover, lines 51–76 discuss a well-known observation: a weaker model can correct ensemble errors while a stronger one can be redundant. The relationship between individual performance, error diversity, and ensemble performance has been studied extensively [4, 5]. Without proper positioning and contextualization, it is hard to identify the main novelties of this work. Furthermore, existing approaches to pool selection and pruning, including greedy selection [1, 6], are neither sufficiently contextualized nor included in the experimental comparisons.
(-) The authors explicitly restrict their results to global, unregularized convex aggregation under squared/Brier loss, so I am not questioning the theorem outside its stated assumptions. However, could a candidate rejected under this particular objective still be useful under a different aggregation scheme, such as alternative linear weighting, nonlinear stacking, or dynamic selection? Previous work demonstrates the importance of the combination mechanism and locally varying classifier competence [2, 7]. What properties still hold under different aggregation schemes? If the screening method is only optimal or useful for a particular combiner and objective, the authors should at least compare its predictive performance and computational trade-offs with established alternatives. This would help assess whether the restriction provides a worthwhile practical advantage or whether the contribution should be positioned more narrowly.
(-) The tabular regression experiments consist of six incumbent predictors (as mention in appendix D, two models from each of three families are trained to compose the pool) with a candidate from the held-out family, while the LLM experiments use seven or eight incumbents. Ensemble selection has long considered much larger and more diverse model libraries. for example, Caruana et al. [1] select from thousands of models generated using different algorithms and parameter settings. While evaluating hundreds of LLMs would be expensive, tabular classification datasets would offer an affordable setting for studying larger pools, using datasets from TabZilla or OpenML suites. Furthermore, using different model families does not itself establish sufficient error diversity. Duin [7] discusses multiple ways of constructing complementary classifiers, including varying training samples and feature representations, while Brown et al. [8] provide a broader categorization of diversity-generation strategies.
(-) In my opinion, varying pool size and diversity through resampling, feature subsets, hyperparameters, and learning algorithms would allow a more controlled assessment of screening efficiency and scalability [1, 7, 8]. I understand that further simulations with LLMs and larger pools may be costly, but tabular regression and classification experiments could provide stronger support for the paper’s main claims. At the current stage, the experiments do not systematically examine redundant or complementary candidates, locally specialized candidates, or different error-correlation structures, despite their importance in ensemble learning [2, 4, 5]. Would these characteristics affect the number of calls needed for rejection? Could candidates rejected under global aggregation nevertheless be useful as local experts under conditional selection?
(-) I also think classical sampling criteria based on uncertainty or disagreement deserve comparison. Predictive entropy for classification or variance across incumbent predictions could provide simple, inexpensive alternatives. Diversity measures such as disagreement, correlation, and vote entropy have long been studied in ensemble learning [4]. Adapting such scores for sampling would help establish whether the proposed method offers significant benefits over simpler criteria. The comparison should retain matched candidate-call
budgets and appropriate sampling weights so that differences reflect the sampling strategy rather than a change in the evaluation target.
(-) Lastly, when using LLMs and LLMs ensembles/models created at different periods/time, a question that raises is whether the daaset used for the experiments were already part of the LLM pre-training data (in other words, data leakage). And whether such possible leakage could have a significant impact in the results. Authors must check this point and at least comment on that as a “limitation of the current study”
Minor points: I think some parts of the introduction could be better explained. Like the Oracle from Table 1 should be properly defined (like, for instance, adding its definition in a footnote), asanding it is not the classical Oracle definition from ensemble learning and that may cause confusion from my underst with readers familiar with the ensemble literature.
I am willing to increase my scores if the authors adequately address these concerns during the rebuttal.
References
[1] Caruana, R., Niculescu-Mizil, A., Crew, G., and Ksikes, A. “Ensemble Selection from Libraries of Models.”
[2] Woods, Kevin, W. Philip Kegelmeyer, and Kevin Bowyer. "Combination of multiple classifiers using local accuracy estimates."
[3]Ko, Albert HR, Robert Sabourin, and Alceu Souza Britto Jr. "From dynamic classifier selection to dynamic ensemble selection."
[4] Kuncheva, L. I., and Whitaker, C. J. “Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy.”
[5] Brown, G., Wyatt, J. L., and Tiňo, P. “Managing Diversity in Regression Ensembles.”
[6] Dai, Q., Ye, R., and Liu, Z. “Considering Diversity and Accuracy Simultaneously for Ensemble Pruning.”
[7] Duin, R. P. W. “The Combining Classifier: To Train or Not to Train?”
[8] Brown, G., Wyatt, J., Harris, R., and Yao, X. “Diversity Creation Methods: A Survey and Categorisation.”
[9] Ong, Isaac, et al. "Routellm: Learning to route llms from preference data.
We thank the reviewer for the time spent on our manuscript. We would like to address the weaknesses and questions raised, also considering the additional experiments we ran.
W1, Theorem 1. We agree that it is the first-order optimality condition on the simplex. We now state it as a lemma and cite the Frank–Wolfe literature. The technical core is the safe-rejection theorem and its corollary (Theorem 2, now Theorem 1, and Corollary 1 in the submission). They reject from partial candidate outputs with all weights refitted, and the rejection is safe, complete and as early as any error-free rule allows. A new appendix extends this certificate to any combiner that is linear in its weights with a convex loss, including stacking.
W1, sampling. Incumbent loss is our own second sampler, not an external baseline. Proposition 1 shows that it ranks regression rows exactly as alignment width does and bounds that width under Brier loss. The tie is therefore the predicted outcome, and Section 5.2 states it. The benefit is over uniform sampling: 2.9 to 3.3 points in regression and 6.5 to 7.8 points on the LLM panels at 64 calls. Both samplers also improve held-out decision accuracy over uniform sampling at 64 calls, by 2.7 to 3.2 points in regression and 3.9 to 4.7 points on unseen LLM tasks, with all intervals excluding zero (Appendix E.5). Label-free surrogate scores such as entropy, margin and disagreement do not match this: in the same allocator at 64 calls, they fall 4.3 to 6.1 points below alignment on unseen LLM tasks and 3.4 points below in regression, and none significantly improves on uniform sampling (new Appendix E.6).
W2, the 83 to 89%. This number measures full evaluation, which is standard practice, not our method. Certified QP has the same held-out accuracy as evaluation without early stopping (Appendix E.5), so the savings cost nothing on held-out data. Always rejecting scores 54 to 70%.
W2 and Q1, when screening matters. Screening is paid for every released model on every task. Deployment is paid only for accepted models, and about 70% of candidate-task pairs on our LLM panels are not useful. Table 1 shows that an oracle would avoid 40 to 93% of these calls. Screening is a substantial share when releases are frequent, tasks are many, or traffic per task is low. The certificate costs 0.34% of the inference it avoids (Appendix D.5). Screening also limits deployment: in the new stream experiment it deploys about 4 models, against 10 for a joint refit and 100 for accepting every model (Table 2).
Q2, greedy against joint. Each acceptance refits all weights jointly. The only greedy step is that rejected candidates are not revisited. Against a joint refit over all 100 classifiers with the same combiner, the screened ensemble keeps 97% of the held-out gain with 4.3 models against 10.1, and 99.5% with a 0.25% threshold (Table 2). Stacking is a different choice of combiner, not an alternative to screening: it needs every candidate's predictions before it can be fitted. The two compose, since the certificate extends to linear stacking (Appendix G).