acceptodds
Under review as a conference paper at ICLR 2027

Screen Before You Add: Reducing Candidate Inference in Model Ensembles

Abstract

As new models become available, existing ensembles must be updated to be competitive, making them intrinsically dynamic. Deciding whether to add a candidate model can require inference on an entire annotated dataset. Such a screening becomes prohibitive for models with costly inference especially when they ultimately add no value. Even under simple convex aggregation, relying on benchmark scores can be misleading as strong standalone performance does not guarantee ensemble improvement, and fitted ensemble weights can hurt test performance. In this work, we study candidate screening and show how early rejection can be achieved safely, exploiting residual geometry for stratification and rejecting the candidate when the best possible remaining predictions cannot achieve the necessary improvement. The key result that justifies this approach is that, under convex aggregation and squared or Brier loss, the population optimum improves if and only if a candidate induces a prediction change positively aligned with the residuals of the population-optimal existing ensemble. An optimistic quadratic program (QP) provides certified early rejection, while posterior-scenario sampling offers an alternative heuristic. We evaluate the approach by showing the tradeoff between saved calls and screening accuracy, which refers to recovering the same accept/reject result as for the entire annotated dataset. We assess empirically whether full-dataset decisions predict candidate usefulness on a held-out test set. On two LLM panels, stratified screening with certified QP rejection saves 21–23% of candidate calls while keeping 100% screening accuracy. In our experiments, sampling posterior scenarios further increases call savings to approximately 37%, with a loss of less than one percentage point in screening accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
ensemble learning, model selection, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GLohOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether to add a candidate model to a convex ensemble without evaluating it on the full screening set. It uses residual information to stratify queries and an optimistic QP to certify that unseen predictions cannot produce a 1% loss reduction. A posterior-sampling extension permits earlier, uncertified rejection. Certified rejection saves 21–23% of candidate calls on the LLM panels while the posterior variant saves about 37%.

Critical strengths

The problem is clearly defined. The optimistic QP is a useful construction because it accounts for refitting all ensemble weights. The main proofs look correct, and the experiments include relevant sampling and stopping baselines. The authors also distinguish screening-set guarantees from held-out performance.

Critical weaknesses
  • The theoretical and methodological contribution is narrower than what is claimed. The paper claims Theorem 1 as one of its three main contributions, but it is just an application of standard convex optimality: a new vertex improves the optimum exactly when it has negative directional derivative. This is standard in the literature, and underpins methods such as Frank-Wolfe. The sampling approach is related to established loss-guided evaluation and surrogate-based stratification, some of which the paper cites. Its alignment score does not consistently outperform the simpler incumbent-loss score, leaving its additional benefit unclear.
  • The practical value of the proposed method is not sufficiently demonstrated. The primary metric measures agreement with the full-screening-set decision, which itself agrees with held-out usefulness in only 83–89% of cases. Furthermore, the real computational bottleneck in ensembling is deployment rather than screening. Hence, savings in screenings may not ultimately make much practical difference.
Questions for authors
  1. Can you explain when screening is a substantial amount of the computational budget?
  2. Your proposed method is "greedy" in the sense that it sequentially determines whether to add a new model to the ensemble. How much performance is lost relative to a joint optimization (e.g. via stacking) over all models?
2 comments

We thank the reviewer for the time spent on our manuscript. We would like to address the weaknesses and questions raised, also considering the additional experiments we ran.

  • W1, Theorem 1. We agree that it is the first-order optimality condition on the simplex. We now state it as a lemma and cite the Frank–Wolfe literature. The technical core is the safe-rejection theorem and its corollary (Theorem 2, now Theorem 1, and Corollary 1 in the submission). They reject from partial candidate outputs with all weights refitted, and the rejection is safe, complete and as early as any error-free rule allows. A new appendix extends this certificate to any combiner that is linear in its weights with a convex loss, including stacking.

  • W1, sampling. Incumbent loss is our own second sampler, not an external baseline. Proposition 1 shows that it ranks regression rows exactly as alignment width does and bounds that width under Brier loss. The tie is therefore the predicted outcome, and Section 5.2 states it. The benefit is over uniform sampling: 2.9 to 3.3 points in regression and 6.5 to 7.8 points on the LLM panels at 64 calls. Both samplers also improve held-out decision accuracy over uniform sampling at 64 calls, by 2.7 to 3.2 points in regression and 3.9 to 4.7 points on unseen LLM tasks, with all intervals excluding zero (Appendix E.5). Label-free surrogate scores such as entropy, margin and disagreement do not match this: in the same allocator at 64 calls, they fall 4.3 to 6.1 points below alignment on unseen LLM tasks and 3.4 points below in regression, and none significantly improves on uniform sampling (new Appendix E.6).

  • W2, the 83 to 89%. This number measures full evaluation, which is standard practice, not our method. Certified QP has the same held-out accuracy as evaluation without early stopping (Appendix E.5), so the savings cost nothing on held-out data. Always rejecting scores 54 to 70%.

  • W2 and Q1, when screening matters. Screening is paid for every released model on every task. Deployment is paid only for accepted models, and about 70% of candidate-task pairs on our LLM panels are not useful. Table 1 shows that an oracle would avoid 40 to 93% of these calls. Screening is a substantial share when releases are frequent, tasks are many, or traffic per task is low. The certificate costs 0.34% of the inference it avoids (Appendix D.5). Screening also limits deployment: in the new stream experiment it deploys about 4 models, against 10 for a joint refit and 100 for accepting every model (Table 2).

  • Q2, greedy against joint. Each acceptance refits all weights jointly. The only greedy step is that rejected candidates are not revisited. Against a joint refit over all 100 classifiers with the same combiner, the screened ensemble keeps 97% of the held-out gain with 4.3 models against 10.1, and 99.5% with a 0.25% threshold (Table 2). Stacking is a different choice of combiner, not an alternative to screening: it needs every candidate's predictions before it can be fitted. The two compose, since the certificate extends to linear stacking (Appendix G).

Thanks for the reply. They partially address some of my concerns, such as regarding the framing of the theorem's novelty. I will raise my score to a 2. However, I am still not convinced by the significance of the paper. In particular, Q1 is not adequately addressed. The example given is just a hypothetical, not a practical scenario.

Reviewer_mpB4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses an important question in building model ensembles: would adding a newly defined model improve performance while also reducing the number of candidate predictions/candidate calls for evaluation.The authors propose a population improvement under squared or multiclass Brier loss through residual alignment, then proposes residual-informed stratification and an optimistic quadratic program for certified early rejection on a finite screening set. A posterior-scenario extension trades certification for additional call savings. Experiments cover 35 tabular regression datasets and a few LLM evaluation tasks. The authors find that informed sampling and certified rejection reduce evaluation calls while preserving full-screening decisions

Critical strengths
  • As evaluating a new model can be expensive, especially with LLMs, I find the idea of reducing the calls needed to decide whether it is worth adding to an ensemble an useful and relevant idea.
  • The authors establish when a candidate can improve the ensemble under their assumptions and use this result to justify early rejection with a good theoretical development.
  • I appreciate that the authors also check candidate usefulness on held-out examples, rather than only measuring agreement with the full-screening decision.
  • Authors provided the source code as supplementary material for reproducibility. Overall it is well documented, which helps with reproducing the experiments.
Critical weaknesses

(-) I found the paper insufficiently positioned within ensemble selection and pool-management research. Section 2 mentions stacking, aggregation, diversity, and safe screening very superficially, but does not sufficiently connect the proposed problem to established selection from model libraries [1], local classifier selection & weighting [2,9], or adaptive pool maintenance [3]. Although these approaches may not tackle exactly the same screening problem, they are important for explaining what is new and which practical ensemble-maintenance decisions the method addresses. Moreover, lines 51–76 discuss a well-known observation: a weaker model can correct ensemble errors while a stronger one can be redundant. The relationship between individual performance, error diversity, and ensemble performance has been studied extensively [4, 5]. Without proper positioning and contextualization, it is hard to identify the main novelties of this work. Furthermore, existing approaches to pool selection and pruning, including greedy selection [1, 6], are neither sufficiently contextualized nor included in the experimental comparisons.

(-) The authors explicitly restrict their results to global, unregularized convex aggregation under squared/Brier loss, so I am not questioning the theorem outside its stated assumptions. However, could a candidate rejected under this particular objective still be useful under a different aggregation scheme, such as alternative linear weighting, nonlinear stacking, or dynamic selection? Previous work demonstrates the importance of the combination mechanism and locally varying classifier competence [2, 7]. What properties still hold under different aggregation schemes? If the screening method is only optimal or useful for a particular combiner and objective, the authors should at least compare its predictive performance and computational trade-offs with established alternatives. This would help assess whether the restriction provides a worthwhile practical advantage or whether the contribution should be positioned more narrowly.

(-) The tabular regression experiments consist of six incumbent predictors (as mention in appendix D, two models from each of three families are trained to compose the pool) with a candidate from the held-out family, while the LLM experiments use seven or eight incumbents. Ensemble selection has long considered much larger and more diverse model libraries. for example, Caruana et al. [1] select from thousands of models generated using different algorithms and parameter settings. While evaluating hundreds of LLMs would be expensive, tabular classification datasets would offer an affordable setting for studying larger pools, using datasets from TabZilla or OpenML suites. Furthermore, using different model families does not itself establish sufficient error diversity. Duin [7] discusses multiple ways of constructing complementary classifiers, including varying training samples and feature representations, while Brown et al. [8] provide a broader categorization of diversity-generation strategies.

(-) In my opinion, varying pool size and diversity through resampling, feature subsets, hyperparameters, and learning algorithms would allow a more controlled assessment of screening efficiency and scalability [1, 7, 8]. I understand that further simulations with LLMs and larger pools may be costly, but tabular regression and classification experiments could provide stronger support for the paper’s main claims. At the current stage, the experiments do not systematically examine redundant or complementary candidates, locally specialized candidates, or different error-correlation structures, despite their importance in ensemble learning [2, 4, 5]. Would these characteristics affect the number of calls needed for rejection? Could candidates rejected under global aggregation nevertheless be useful as local experts under conditional selection?

(-) I also think classical sampling criteria based on uncertainty or disagreement deserve comparison. Predictive entropy for classification or variance across incumbent predictions could provide simple, inexpensive alternatives. Diversity measures such as disagreement, correlation, and vote entropy have long been studied in ensemble learning [4]. Adapting such scores for sampling would help establish whether the proposed method offers significant benefits over simpler criteria. The comparison should retain matched candidate-call budgets and appropriate sampling weights so that differences reflect the sampling strategy rather than a change in the evaluation target.

(-) Lastly, when using LLMs and LLMs ensembles/models created at different periods/time, a question that raises is whether the daaset used for the experiments were already part of the LLM pre-training data (in other words, data leakage). And whether such possible leakage could have a significant impact in the results. Authors must check this point and at least comment on that as a “limitation of the current study”

Minor points: I think some parts of the introduction could be better explained. Like the Oracle from Table 1 should be properly defined (like, for instance, adding its definition in a footnote), asanding it is not the classical Oracle definition from ensemble learning and that may cause confusion from my underst with readers familiar with the ensemble literature.

I am willing to increase my scores if the authors adequately address these concerns during the rebuttal.

References

[1] Caruana, R., Niculescu-Mizil, A., Crew, G., and Ksikes, A. “Ensemble Selection from Libraries of Models.”

[2] Woods, Kevin, W. Philip Kegelmeyer, and Kevin Bowyer. "Combination of multiple classifiers using local accuracy estimates."

[3]Ko, Albert HR, Robert Sabourin, and Alceu Souza Britto Jr. "From dynamic classifier selection to dynamic ensemble selection."

[4] Kuncheva, L. I., and Whitaker, C. J. “Measures of Diversity in Classifier Ensembles and Their Relationship with the Ensemble Accuracy.”

[5] Brown, G., Wyatt, J. L., and Tiňo, P. “Managing Diversity in Regression Ensembles.”

[6] Dai, Q., Ye, R., and Liu, Z. “Considering Diversity and Accuracy Simultaneously for Ensemble Pruning.”

[7] Duin, R. P. W. “The Combining Classifier: To Train or Not to Train?”

[8] Brown, G., Wyatt, J., Harris, R., and Yao, X. “Diversity Creation Methods: A Survey and Categorisation.”

[9] Ong, Isaac, et al. "Routellm: Learning to route llms from preference data.

Questions for authors
  1. Can the authors explain how their contribution relates to established ensemble selection, accuracy–diversity-based pruning as well as local classifier weighting? What does the proposal offer beyond the existing literature ? and what are its limitations?

2)Could a candidate rejected under the proposed criterion still improve the ensemble when using another combination mechanism, such as nonlinear stacking or dynamic classifier selection? Moreover, what evidence justifies choosing the current aggregation scheme and squared/Brier loss over alternatives?

  1. Can the authors compare their approach with classical greedy selection or accuracy–diversity-based pool management? comparing predictive performance, candidate calls, and computational overhead would help establish whether the proposed restriction provides a worthwhile practical advantage.

  2. Can the authors provide a more controlled evaluation with larger and more diverse pools ? Varying pool size, error correlation (diversity metrics among base models), and the presence of redundant or locally specialized candidates would help explain when screening is effective and whether its computational benefits persist in different scenarios. Without such experiments it is hard to fully understand the main benefits of the current approach as well as its current limitations.

3 comments

We thank the reviewer for the comprehensive feedback. We have added new experiments both in the main paper and the appendix to address some of the points raised, updated the related work section, clarified the contribution, and extended the theory beyond the convex setting to more general aggregation, as we only rely on scores being linear in the weights and the loss being convex in the scores.

  • Q1. Library selection, pruning, dynamic selection and routing start from a pool whose predictions are already computed. We decide whether to pay for a candidate's predictions, with a certificate that reproduces the full-evaluation decision, and those methods can then run on the accepted pool. We added a Related Work paragraph that states this and cites the suggested references. Limitations: our experiments use global convex weights under squared or Brier loss, savings arise only on rejected candidates, and candidates that help only locally are outside the criterion. The Conclusions now state the combiner and locality limits.

  • Q2. Yes. A richer combiner searches a larger class, so it can use a candidate our rule rejects. To address this we have extended the theory and added some further experiments. Proposition 6 in the new Appendix G shows that the certificate does not need convex weights or Brier loss: safe rejection extends to any combiner that is linear in its weights with a convex loss, such as log-opinion pools and linear stacking, and it made no wrong rejection in 14,220 replayed screening paths. It is weaker there, certifying 51% and 23% of rejections early for log pools against 74% for the convex pool on the LLM cases, and nonlinear stacking and dynamic selection remain outside the guarantee. We choose convex aggregation and Brier loss inspired by two practical precedents for this setting.

  • Q3. Greedy selection with replacement on the same loss converges to the same convex fit, so predictive performance is identical. The difference is in calls: selection needs every candidate prediction, while Certified QP reaches the same decision with 21 to 23% fewer calls (Section 5.3), at an overhead of 0.34% of the inference time saved (Appendix D.5).

  • Q4. We added a controlled stream with a pool of 100 classifiers from varied model families, on 6 OpenML-CC18 datasets with 20 seeds each (Table 2). The computational benefit persists and grows with the pool: Certified QP saves 23% of candidate calls after 10 arrivals, 34% after 50 and 37% after 98, and this growth holds in each of the 6 datasets. Every decision still matches full evaluation, whereas random subsampling with the same number of rows gets 0.7% of decisions wrong. About 94% of arrivals add less than 1% to the ensemble, and these redundant candidates are the ones the certificate rejects early.

  • Leakage. Contamination in model pretraining cannot be ruled out, as Appendix D.6 states. It does not affect our main claims: screening accuracy and calls saved are measured against full evaluation on the same data, so the certificate holds whatever the models have seen. It could change which models appear useful on these benchmarks. We now state this as a limitation in the Conclusions.

  • oracle definition. we added a clarification in Table 1 caption.

  • Classical sampling criteria. We plugged entropy, margin and disagreement scores into the same stratified allocator, at matched budgets and with the same estimator. At 64 calls they fall 4.3 to 6.1 points below alignment on unseen LLM tasks and 3.4 points below in regression. On the unseen-model panel the gap has the same sign but is not significant. We added this comparison as a new appendix subsection (Appendix E.6).

Reviewer_mpB4OpenReview ↗

Response to rebuttal

Thanks to the author for providing an answer to my points and adding more information regarding ensemble learning literature to help position the paper, as well as additional information regarding data leakage in LLMs moved into the main manuscript.

There are still a few points that, in my opinion, were not properly addressed.

i) The new analysis increasing pool size is a good step in the right direction, but it still lacks coupling it with a diversity analysis. From my understanding, just generating 100 models cannot guarantee properties (nor how does the system behaves under different conditions). Using different model families does not necessarily mean that they make complementary errors. Also, the fact that 94% of arrivals fall below the 1% improvement threshold does not establish that these candidates are redundant. ii) Still think the scope of the work is quite limited given all constraints needed to make it work and the fixed ensemble as condition and restriction on fusion scheme/losses. I am still not convinced that, after the arrival of a new model, it may be beneficial to not only decide whether to add it to the previous set, but doing a new full updated that could include removing other models. This is also a concern shared by other reviewers (reviewer M2k1).

iii)Furthermore, I also think the paper lacks a comparison with other strategies in ensemble learning literature (based not restricted to the same loss) to manage a pool of models to demonstrate its relevance. The methods based on different heuristics (like predictive performance and diversity or greedy search) are not limited to the same objectives. And a comparison against different heuristics that are based on different optimization objectives outside of the ones considered in this paper should be made. Because without it, in my opinion, the value of focusing on these specific cases is not well justified.

We thank the reviewer for the further feedback and take this opportunity to clarify our results and scope.

(i) Diversity and the guarantee. Lemma 1 (formerly Theorem 1) characterizes exactly when a candidate improves population risk under convex aggregation and squared/Brier loss. Appendix G extends the certificate to any combiner that is linear in its weights with a convex loss. The 1% threshold only sets how much improvement warrants an addition. It is a free parameter, and Table 2 also evaluates 0.25%. We agree that "below threshold" is more precise than "redundant." Studying error correlation could explain when screening saves more calls. However, bounding those savings is a statistical question that our work does not target. Our guarantee concerns the correctness of each early rejection on the given screening set. It holds for any error-correlation structure, at the cost of being conservative.

(ii–iii) Scope within ensemble updates. Our contribution is the atomic "add" operation within a broader ensemble update. The update rule defines what counts as a useful addition, and our certificate reduces the calls needed to evaluate that decision under the stated assumptions. We make no assumption on the incumbent set, which can change at every arrival. In Table 2, each acceptance triggers a refit of all active weights, and 291 of 690 accepted arrivals displaced an incumbent. The joint-refit baseline also evaluates the full update you propose by reconsidering every model seen so far. Comparing screening with full evaluation under the same rule isolates our contribution: across 120 streams, it preserved every decision with 37% fewer candidate calls.

Our approach does not replace selection, pruning or other pool-management methods. It complements them. The update rule defines what counts as a useful addition, and screening reduces the cost of evaluating it. Practitioners remain free to manage the accepted pool as they see fit. Comparing against heuristics that optimize other objectives would compare update rules, not screening methods.

Reviewer_M2k1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether a new model can improve a convex ensemble while reducing the candidate predictions needed for the decision. It uses residual alignment to guide stratified sampling and an optimistic QP to certify early rejection under squared or Brier loss. A posterior extension trades some screening accuracy for additional savings. Experiments cover 35 regression datasets and 12 LLMs on 15 classification and multiple-choice tasks.

Critical strengths
  1. The optimistic QP accounts for both missing candidate predictions and refitting all ensemble weights. Under the stated prediction bounds, it certifies early rejection without discarding a candidate that full evaluation would accept. This gives a clear guarantee for reducing screening cost.

  2. The experiments separate sampling from stopping and show measurable savings. On the two LLM panels, certified QP screening saves 21–23% of candidate calls while matching every full-screening-set decision at the full budget. This supports the usefulness of the method for its stated single-candidate screening task.

Critical weaknesses
  1. The scope is narrow: the paper screens one candidate against a fixed ensemble, aiming to reproduce the full-evaluation decision with fewer candidate calls. It does not evaluate repeated updates as new models arrive. I would like to see how the number of active models, test performance, and deployment inference cost change throughout this process. Although refitting can give old models zero weight, the method does not explicitly control ensemble size or deployment cost. Such an experiment would show whether screening savings come with a growing deployment burden, and whether additional pruning or compression is needed. Given the motivation of maintaining dynamic ensembles, I would like the paper to broaden its scope to address how best to update an existing ensemble when a new model arrives: add it, replace one existing model, or replace several, while accounting for both performance and deployment cost.

  2. In Section 5.1, the distinction between the “unseen-task” and “unseen-model” evaluations is unclear. The former tests each of the original eight models against the other seven, while the latter tests four additional models against the original eight; both evaluate adding one candidate to an existing ensemble. Table 4 describes the four new models as “held out from development,” but the paper does not clearly explain whether or how the original eight models were used to choose the screening rules or hyperparameters, on which tasks, or which choices were fixed before evaluating the new models. Without these details, it is hard to tell what the second panel establishes beyond testing additional model combinations.

Questions for authors

Please refer to the weaknesses.

5 comments

We thank the reviewer for the feedback. We would like to highlight a few points on the paper and raise some additional elements that we obtained from new experiments inspired by the comments of the reviewer.

  • W1. We added a stream experiment in which 98 classifiers arrive one at a time at a deployed ensemble (Table 2), tracking active models, held-out performance and screening cost throughout. Certified QP makes the same decision as full evaluation in all 11,760 arrivals while saving 37% of the calls. The 1% rule ends with about 4 models, against 10 for a joint refit over all arrivals and 100 for accept-all, while keeping 97% of the joint refit's held-out gain. With a budget of 3 models, where each arrival must replace an incumbent, it still keeps 96% of the joint refit's gain.

  • W2. Section 5.1 and Appendix D.2 define both. The first adds each original model to the other seven on 15 task families held out from development. The second adds each of four new-family models to the fixed eight on the same tasks. The goal was to validate the method first on tasks and then on models that were not used during its development.

Reviewer_M2k1OpenReview ↗

Follow-up questions on W1

Thank you for adding the sequential-arrival experiment. However, the current comparison contrasts a sequential procedure using a 1% acceptance threshold with a joint refit without a comparable ensemble-size constraint. Retaining 97% of the gain with about four models therefore does not establish whether the procedure offers a competitive trade-off between predictive performance and ensemble size.

I would like to see performance–ensemble size curves obtained by varying the acceptance threshold or target ensemble size, with comparisons against representative ensemble selection methods and a simple post-hoc pruning baseline. Specifically, the pruning baseline could first fit ensemble weights over the full model pool, retain the top K models with the largest weights, and then refit the weights using only those selected models. Varying K would produce the corresponding curve.

These comparisons would allow us to assess test performance at the same ensemble size, or the number of models needed to achieve comparable test performance, and better evaluate the procedure’s practical value for ensemble maintenance.

Thank you for the follow-up. We ran the comparison you describe on the same 120 streams and added it as a new appendix subsection with a figure (Appendix E.7).

  • Baselines. Top-K pruning as you specify it: convex weights over all 100 models, keep the K largest, refit. Greedy ensemble selection (Caruana et al., 2004), capped at K distinct models. For our rule we vary the acceptance threshold from 4% to 0.1%.
  • You are right about the threshold. At equal size, top-K pruning is better than our sequential rule. At the size of the 1% rule (4.3 models) its held-out Brier loss is 0.5% lower [0.3, 0.8], and the gap grows to 2.4% at a 4% threshold. The threshold is not a good control for ensemble size, and we now say so in Section 5.5.
  • Screening composes with pruning. Both baselines choose after every model has been queried on every screening row. Screening decides which of those calls to make. We therefore also prune the ensemble that Certified QP deploys at a 0.25% threshold to its K largest weights. This adds no candidate call, because accepted models are already fully queried. At every K the result is statistically indistinguishable from top-K pruning of the full pool, within 0.05% in held-out Brier loss for K ≥ 4, with 31% fewer candidate calls.

Our contribution is the cost of the add decision, not a new selection rule. The new figure (Figure 5, in Appendix E.7) shows the two side by side: the same size–quality curve as post-hoc pruning, reached with fewer candidate calls.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. The revision makes five changes.

  • Arrival experiment (new Section 5.5, Table 2). To test dynamic ensembles, we added an experiment where a stream of 100 classifiers arrives one at a time at a deployed ensemble, on 6 OpenML-CC18 datasets. Certified screening matches full evaluation at every arrival and saves 37% of candidate calls. It keeps 97% of a joint refit's held-out gain with about 4 deployed models instead of 10. A size-budget variant covers replacing an incumbent.
  • Beyond convex aggregation (new Appendix G). The safe-rejection certificate extends to any combiner that is linear in its weights with a convex loss, including linear stacking and logarithmic pools. We prove this and validate it on the paper's cached predictions, with no unsafe rejection.
  • Positioning (Section 2). A new paragraph relates screening to ensemble selection, pruning and dynamic selection. These methods act on a pool whose predictions are already computed. Screening is the preceding step and composes with them.
  • Scope of the claims (Sections 3 and 6). The residual-alignment criterion is now stated as a lemma and linked to the Frank–Wolfe literature. The conclusions state the limitations: richer combiners certify fewer rejections, and candidates that help only locally remain outside the criterion.
  • Stronger sampling baselines (new Appendix E.6). We tested label-free scores such as entropy, margin and disagreement as strata in the same allocator. At 64 calls they fall 4.3 to 6.1 points below alignment on unseen LLM tasks and 3.4 points below in regression, and none significantly improves on uniform sampling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.