- Rating
- 4
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
A theoretical analysis of sample complexity of a learner which is evaluated using pass@k metric is conducted. This is compared to existing results for pass@1, under both standard and robust evaluation. In contrast to previous work, the demonstrator is allowed to be suboptimal.
Critical strengths
- The new bounds with pass@k show that the how the sample complexity class improves with introduction of pass@k i.e. instead of , which is useful for settings where pass@k might be used in place of pass@1. Not all problem settings allow use of pass@k, but many do, so pass@k analysis remains important.
- A suboptimal demonstrator has not been considered in prior work, which is a very practical consideration as demonstrator's are rarely optimal in real world. This work allows the demonstrator to be arbitrarily suboptimal.
Critical weaknesses
- All of these bounds depend on which can significantly limits the applicability of these results. For continuous rewards or continuous or these results are not applicable, but even for binary rewards, these might become vacuous very quickly in some settings.
However, this is at least partly commented upon near the end of the paper.
Minor Errors
- line 138. A leftover 't'
- line 348. says "Figure" instead of "Algorithm"
Questions for authors
None
We thank the reviewer for the useful and insightful comments. We will correct the minor errors you suggested and include them in the camera-ready version of this paper.