- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposed a small but well curated dataset of 1,200 images that spans a variety of different challenging visual conditions for human pose estimation.
The paper is able to show that the ranking of state of the art models changes depending on the visual category and severity of the condition - this indicates existing benchmark that does not focus on the finegrained categorical performance of the pose estimator can not identify the best/most robust method.
Most interestingly to me, the paper examined the impact of image restoration methods as preprocessing on the pose estimator. They seem to have little effect if any, and depends a lot on the exact method being used and the exact pose estimator.
Critical strengths
The data collection process is rigorous in its design and well explained in the paper. The dataset diversity and quality is extensively evaluated and presented quite clearly.
The conclusions are well supported by the analysis and final experiments. The experiments are also extensive and well demonstrates the value of the dataset as a robustness benchmark.
Critical weaknesses
Since the methodology for creating the dataset is not novel and does not scale up easily, the paper's contribution is judged on the value of the dataset itself.
The dataset targets a small set of robustness relevant conditions: haze/fog, low-light/night, precipitation, glare/backlight, motion blur, focus blur. Since this selection of specific conditions is not well motivated, it is not clear to me how big of an impact this would be for the human pose estimation community.
Even if it is useful to understand the robustness of human pose estimation in these specific conditions, it is not clear how this analysis will improve and drive further algorithm development. The obvious thought is that we should scale up data collection/curation to target these less robust conditions. However this not very insightful to the learning community as a whole. The paper should demonstrate how these insights can be used to benefit model development beyond just data gathering to address these specific scenarios.
Questions for authors
Did I miss insights from the paper that goes beyond identifying the current data gap for training open source models?
The section on the image of restoration is interesting as an initial experimental validation that such methods don't make much difference, but it does not provide any insight on why this is the case. Further analysis on this question would have been more impactful to the image restoration field and better demonstrate the value of this dataset.