- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents an extension of a recent work for eliciting the trade-off weights of a constrained optimization problem. They introduce a sampling technique to select pair of candidates to present to the user that maximize disagreement between the current population of candidate weights (that is iteratively refined based on feedback).
Critical strengths
The proposed approach is simple and sensible, disagreement reranking is a sensible measure of uncertainty, albeit its relationship with standard measures of information like EVOI should be better investigated.
The experimental evaluation is extensive and carefully designed and covers different domains.
Critical weaknesses
The authors present in the motivation combinatorial settings as settings where assumptions of active learning (candidate alternatives available beforehand) do not hold, but then introduce a method that relies on a pool of candidate alternatives available beforehand. This sounds confusing, I believe the setting where the technique is expected to operate and its assumptions should be clarified.
Results do not indicate clear advantages over BALD and EVOI, which are principled preference learning alternatives. The authors do recognize the lack of significant advantages in the section devoted to the comparison, but the abstract states "showcase that it reduces both end-regret and the number of queries compared to the state-of-the-art.". I imagine this refers to the comparison with Defresne et al, the algorithm on top of which the approach is developed, but this should be made clearer to avoid overclaiming. In any case, BALD and EVOI and natural competitors and a lack of advantage wrt them substantially weakens the contribution.
There is a problem with the formatting of references, many of them only report authors, title and year. Not critical, but should be addressed.
Questions for authors
Aren't candidate alternatives assumed not to be available in combinatorial settings? can you clarify the setting where the approach operates?
Can you clarify the reasons why the proposed approach should be preferred wrt techniques like EVOI maximization?
2 comments
I would like to thank the authors for the clarifications. However, my concerns are mostly unaffected.
The introduction, that hasn't been changed despite the possibility to upload a revised version, still mentions the motivation of combinatorial settings where assumptions of active learning (candidate alternatives available beforehand) do not hold. This is misleading.
The argument that BALD and EVOI cannot be computed for non-Bayesian models, while DRES can, is not entirely convincing, as the experimental evaluation applies all of these methods to the same ensemble particles, and none of them is exact for non-Bayesian ensembles.
We thank the reviewer for pointing out the framing issues.
To address question 1, we develop our analysis in the setting where a solution pool of either pareto/feasible/relaxed can be pre-computed. We rely on solution enumeration, where DRES essentially replaces solver calls with an argmax. The final proposed solution is constructed using a solver call. In settings where precomputation is impossible, Dragone et al. (2018) is preferred.
For question 2, our methodological contribution is a query selection strategy that makes it possible to compare classes of uncertainty models, probabilistic or not. It does not aim to replace BALD or EVOI, which cannot be computed exactly for the non-Bayesian models central to our study; instead, it enables comparisons over a broader class of uncertainty models. The comparisons to BALD and EVOI are therefore not part of our primary contribution, which we will clarify in the revised manuscript.