- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to select a small ensemble of complementary candidates using limited binary correctness or pairwise preference queries on tasks drawn from an unknown distribution. It leverages binary feedback and conditional greedy to evaluate potential additions on tasks that are missed by the current ensemble. With pairwise feedback, the proposed method optimizes performance against the worst individual rival. It conducted experiments on controlled synthetic populations to demonstrate the effectiveness of the proposed method.
Critical strengths
Strengths
- Well-motivated problem. The paper studies candidate complementarity under limited evaluation budgets.
- Careful query accounting. The analysis counts the cost of finding tasks missed by the committee.
- Broad theoretical analysis. The results cover approximation guarantees, computational hardness, and statistical lower bounds.
- Transparent experiments. The paper reports unfavorable results and clearly states the experiments’ limitations.
Critical weaknesses
Weaknesses
-
The distinction from prior methods could be made clearer. Sections 1.1 and 1.2 summarize the contributions and related work, but it remains difficult to assess how the proposed methods advance beyond noisy submodular maximization and action-elimination approaches. Please identify the specific technical challenges that these methods do not directly address, and explain which components of the proposed algorithms or analyses address each challenge. A more explicit comparison of feedback assumptions, query accounting, sample reuse, and guarantees would help clarify the novelty.
-
The agreement-based query savings remain local. The disagreement test is analyzed for comparing two candidate additions, whereas conditional greedy (CG) selects among multiple candidates using a separate elimination procedure. The paper does not show whether incorporating the disagreement test into CG could reduce the query cost of full ensemble selection. A discussion of the technical challenges would clarify the scope of this result; if feasible, an experiment comparing such an extension with CG and the screened baselines would further strengthen the paper.
-
The evidence for practical relevance is limited. The paper acknowledges that its experiments use synthetic populations and that deployment considerations remain open. A small evaluation on real model outcomes would help assess whether the observed benefits of conditional sampling arise in practice. The paper would also benefit from discussing how its assumptions of strict rankings and uniform query costs affect settings with noisy or tied judgments, or with unequal evaluation costs. Finally, since answer selection is outside the paper’s scope and the lottery guarantee averages over committee draws, it would help to specify which deployment settings the guarantee is intended to support.
Questions for authors
-
What technical obstacles/differences distinguish this approach from the closest noisy submodular maximization and robust optimization methods?
-
What prevents the disagreement test from being used in complete ensemble selection?