- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces TULIP-Bench, a Parkinson’s disease movement dataset that combines three types of data together. The dataset includes 66 patients performing hand and gait tasks, with 10 patients recorded under both DBS OFF and ON conditions. The authors also establish several benchmark tasks to evaluate 3D pose estimation, clinically relevant movement features, and treatment response detection. Their experiments reveal some interesting findings: lower pose estimation errors do not necessarily lead to more accurate clinical movement features, and the importance of 3D pose estimation varies across different tasks. They also show that continuous movement features can capture DBS-related changes that are not reflected in conventional UPDRS scores.
Critical strengths
One strength of this paper is that it brings together three complementary types of data, including multi-view videos, 3D poses, and clinical assessments with DBS treatment responses, which makes the dataset quite valuable and distinctive. The authors also evaluate a wide range of existing models, such as VideoPose3D, MotionAGFormer, PoseFormerV2, WHAM, and SAM 3D Body, and conduct extensive experiments to demonstrate the usefulness of the benchmark. More importantly, these experiments lead to several interesting findings: (1) lower pose estimation errors do not necessarily mean more accurate clinical movement features, which is crutial since this dataset is meant to help implementation of different models to clinical environments; (2) the importance of 3D pose estimation depends on the task, as 2D poses can be sufficient for fist severity classification but not for accurate movement feature extraction; and (3) continuous movement features can reveal DBS related changes that conventional UPDRS scores may miss. I think these findings are meaningful and provide useful directions for future research on clinically relevant pose estimation.
Critical weaknesses
My main concern is the reliability of the 3D ground truth. The authors generate it through multi-view triangulation and additional optimization, but I would like to see a clearer discussion of how accurate these references are.
I also have some concerns about the DBS experiments. The treatment cohort includes only 10 patients, and all of them were recorded in the OFF-then-ON order. This could introduce order effects, such as fatigue or familiarity, which should at least be acknowledged as a limitation. The analysis of 18 gait features could also benefit from clearer handling of multiple comparisons. In addition, the responder prediction accuracy of 60% is not particularly convincing, so I think the conclusions about treatment response prediction should be more cautious.
Questions for authors
-
How sensitive are the reported clinical feature errors to potential inaccuracies in the triangulated 3D ground truth? Could these inaccuracies affect the comparisons between different pose estimation methods?
-
Given that all 10 DBS patients were recorded in the OFF-then-ON order, how do the authors account for potential order effects, such as fatigue or familiarity? Also, would the main findings remain significant after correcting for multiple comparisons across the 18 gait features?