- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
the paper studies the problem about how much a selector from second stage can improve best of n sampling given a shortlist. the paper includes theoretical part and empirical part. on the theoretical side, it proposes a split formula to split the expected regret of best of n into three parts. on the empirical part, the paper designs a wide range of experiment setup for multiple datasets and models to measure the called recoverable opportunity.
Critical strengths
- the paper presents an interesting formula to split the expected regret into three different parts, this potentially can be useful for this type of problems.
- the paper has a substantial set of experiments including a wide range of datasets and models, with good mix and balance.
Critical weaknesses
- the related work is more like purely a list of paper, but without very clear statement or explanation on why they are related. some of them seems odd, for example Hron et al., 2021. this is a paper about recommendation system quite far from the topic of this paper I believe. the very weak connection might be it is common in recommendation system, the stagewise selection and ranking framework is used. if that is the reason this paper has to be listed as related work, it should be explained with more clear intention.
- some terms are not defined before used, such as oracle ceiling, it is formally defined in sec 3 but used several times before it. writing problems like this give quite a lot of difficulty for reading frankly.
- the main contributions as summarized in section 1 have a few very concrete numbers, such as 87.7, 9.2, tho being very concrete, it is very hard to understand at the first place what are the general results they deliver, if not further explained, same problems appear in abstract. also as the main theorem in sec 3, thm 3.2 isn't given much explanation or comparison to show how tight the bound is. btw, the theorem statement has math typo, the parentheses of the two expectations are not consistent, they appear E| and E[ right now.
Questions for authors
the best intuitive takeaway i can get is if a shortlist is narrow, then there isnt much room for improvement, if the shortlist is wider, then there could be larger room, and it leaves to the reranking stage if the room can be achieved. can the authors confirm if the takeaway here is correct? if so, can the author make it clear why it isn't an obvious result?