- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper claims to devise a method evaluating the soundness of a benchmark without having to train models to solve it.
Critical strengths
None
Critical weaknesses
Presentation:
- The paper is poorly written and hard to follow.
- A lot of concepts are mentioned without prior definition (even informal). For example I had to read the code to find the definition of what a template is.
- The abstract contains too much uninformative jargon does not inform the reader on the problem solved, the method used or the main experimental results.
- In addition to not providing any insight on the methods used in the paper, propositions are basic results of maximums over sets and subsets.
- The experiments are poorly described: the datasets are not presented, the method is not described, the results are large tables whose columns are not presented, there is no clear experimental takeaway.
Contributions:
- The fact that some benchmarks can be solved without solving the task they are supposed to evaluate has been known for quite a while, in different domains (privacy leakage, robotics, question answering...). This paper does not provide additional insights on that phenomenon.
Questions for authors
None.