- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper presents a method to extract ordered information from leaderboards. Since the concept is not explicitly defined in the text, I assume a leaderboard refers to a ranking table evaluating ML models. The items extracted from these leaderboards are organized into a contradiction-free partial order.
However, the paper is difficult to read. The problem must be formally defined, ideally by using a running example within the 9 pages of the main paper. The core problem needs to be fully understandable just by reading the main text (especially given that the main paper plus appendices spans 83 pages!). Furthermore, it is difficult to discern how this approach differs from existing research in the domains of rank aggregation, consensus ranking, or Kemeny consensus. Overall, there is a strong need to clearly define the problem. As it currently stands, it is nearly impossible for a researcher outside this specific field to understand the exact problem being solved and to appreciate any potentially original results.
Critical strengths
It is difficult to identify the main strengths of the paper when the core problem is not clearly defined. One potential strength, however, is the rich dataset provided for evaluation, which is thoroughly documented in the extensive appendices.
Critical weaknesses
The paper is difficult to read. The problem must be formally defined, ideally by using a running example within the 9 pages of the main paper. The core problem needs to be fully understandable just by reading the main text (especially given that the main paper plus appendices spans 83 pages!). Furthermore, it is difficult to discern how this approach differs from existing research in the domains of rank aggregation, consensus ranking, or Kemeny consensus.
Questions for authors
Overall, there is a strong need to clearly define the problem. As it currently stands, it is nearly impossible for a researcher outside this specific field to understand the exact problem being solved and to appreciate any potentially original results.