acceptodds
Under review as a conference paper at ICLR 2027

Multi-perspective videos and 3D poses capturing multi-task Parkinson’s disease assessments and deep brain stimulation treatment effects

Abstract

Quantitative motor analysis is increasingly central to Parkinson's disease (PD) research and has the potential to transform PD clinical care. However, the field remains bottlenecked by datasets targeting coarse, manually-annotated PD severity scales rather than accurate PD motor feature identification or patient outcome prediction. Here, we introduce TULIP-Bench, the first PD movement dataset and benchmark to combine raw multi-perspective videos, triangulated markerless 3D pose ground truth, both fine and gross motor tasks with clinical ratings (bilateral hand movements and gait), and within-patient treatment responses to deep brain stimulation (DBS). The TULIP-Bench benchmark comprises tasks evaluating pose lifting and direct single-camera 3D pose tracking approaches in terms of clinical kinematic feature errors, quantities we found to be weakly associated with standard geometric (millimeter) error measures. We identified a critical perspective-dependent performance gap between single- and multi-camera PD assessments and illustrate how conventional PD clinical scores are unable to resolve DBS-induced gait changes. We also developed a self-supervised cross-cohort model for DBS response scoring based on representations learned from non-DBS PD patients. TULIP-Bench will catalyze a new generation of methods that emphasize clinical deployability, sensitive treatment-focused measurement, and patient outcome prediction.

Primary area:
datasets and benchmarks
Keywords:
Parkinson's disease, multi-perspective, 3D, explainable AI, digital medicine, movement analysis, clinical assessment, computer vision, neurostimulation, treatment, DBS

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Q1XFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TULIP-Bench, a Parkinson’s disease movement dataset that combines three types of data together. The dataset includes 66 patients performing hand and gait tasks, with 10 patients recorded under both DBS OFF and ON conditions. The authors also establish several benchmark tasks to evaluate 3D pose estimation, clinically relevant movement features, and treatment response detection. Their experiments reveal some interesting findings: lower pose estimation errors do not necessarily lead to more accurate clinical movement features, and the importance of 3D pose estimation varies across different tasks. They also show that continuous movement features can capture DBS-related changes that are not reflected in conventional UPDRS scores.

Critical strengths

One strength of this paper is that it brings together three complementary types of data, including multi-view videos, 3D poses, and clinical assessments with DBS treatment responses, which makes the dataset quite valuable and distinctive. The authors also evaluate a wide range of existing models, such as VideoPose3D, MotionAGFormer, PoseFormerV2, WHAM, and SAM 3D Body, and conduct extensive experiments to demonstrate the usefulness of the benchmark. More importantly, these experiments lead to several interesting findings: (1) lower pose estimation errors do not necessarily mean more accurate clinical movement features, which is crutial since this dataset is meant to help implementation of different models to clinical environments; (2) the importance of 3D pose estimation depends on the task, as 2D poses can be sufficient for fist severity classification but not for accurate movement feature extraction; and (3) continuous movement features can reveal DBS related changes that conventional UPDRS scores may miss. I think these findings are meaningful and provide useful directions for future research on clinically relevant pose estimation.

Critical weaknesses

My main concern is the reliability of the 3D ground truth. The authors generate it through multi-view triangulation and additional optimization, but I would like to see a clearer discussion of how accurate these references are.

I also have some concerns about the DBS experiments. The treatment cohort includes only 10 patients, and all of them were recorded in the OFF-then-ON order. This could introduce order effects, such as fatigue or familiarity, which should at least be acknowledged as a limitation. The analysis of 18 gait features could also benefit from clearer handling of multiple comparisons. In addition, the responder prediction accuracy of 60% is not particularly convincing, so I think the conclusions about treatment response prediction should be more cautious.

Questions for authors
  1. How sensitive are the reported clinical feature errors to potential inaccuracies in the triangulated 3D ground truth? Could these inaccuracies affect the comparisons between different pose estimation methods?

  2. Given that all 10 DBS patients were recorded in the OFF-then-ON order, how do the authors account for potential order effects, such as fatigue or familiarity? Also, would the main findings remain significant after correcting for multiple comparisons across the 18 gait features?

Reviewer_w4GMOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The authors present a benchmark consisting of hardware-synchronized multi-view videos of patients with Parkinson's disease (PD). The videos are short segments showing two parts of a PD motor examination called UPDRS. While the examination has several components, the presented dataset contains only two parts of it which are walking (UPDRS 3.10) and opening and closing of the hand/fist (UPDRS 3.5 for both left and right hand). The share anonymized data of 56 patients with clinical scores (Track 1) and also paired anonymized data of 10 patients with their deep brain stimulation (DBS) device on and off (Track 2). They also propose 3 tasks and share experimental results for each task. Task 1 is about monocular 2D to 3D pose lifting and Task 2 is about estimating the UPDRS part scores (both related to Track 1 data). Task 3 is about detecting DBS on vs off using Track 2 data.

Critical strengths
  • The dataset can be a useful resource for the community.
  • The data collection and data curation process is clear and well explained and makes sense. Multi-view cameras are synchronized with a wired trigger. Data cleaning (for example for multi-view 3D pose calculation) appears to have been done carefully and with manual consistency check and human inspection.
  • The authors examine a good and relatively up-to-date list of model, for example for monocular pose tracking and 2D to 3D pose lifting. It would have been good if some of the newer models like Multi HMR 2 were also included, but those only came out a few months before the ICLR deadline and I would not expect the authors to have tried them, but perhaps something to consider for future work.
  • Tables presented in the appendices add useful information to the paper for example on the distribution of UPDRS 3.5 and 3.10 scores, hand and gait features used, etc.
Critical weaknesses
  • The authors motivate the need for a video dataset and benchmark by highlighting that existing datasets (one of which is larger than the one being presented here, Table 1) do not include video and only contain 3D pose or SMPL mesh. They then, however, proceed to perform only analysis that use pose (2D or 3D) and do not include any analysis that involves video. If pose or mesh is sufficient, what is the need for a new dataset when an existing multisite dataset in a harmonized SMPL mesh already exists?

  • In addition to 1) including videos, the authors also list 2) accurate 3D pose (from 6-view synchronized and calibrated cameras) and 3) the inclusion of motor tasks beyond walking/gait as their contributions (paragraph starting on line 052). However, However, some of the existing datasets, including some of ones included in the multisite harmonized set, include accurate motion capture pose sequences. Also, the proposed benchmark includes only one more motor assessment video (fist opening and closing) in addition to gait. (3.5 and 3.10 instead of only 3.10)

  • I find the results in Table 5 very confusing and counter intuitive. How can a model achieve an AUC-ROC of 0.83 when Table 6 (the last two rows) show the distribution of the DBS on and off scores is nearly identical?

Questions for authors

minor comments and questions: (in addition to critical ones listed above)

  • is there overlap between track 1 (56) and track 2 (10)? Does 56 include the 10 or is it 56 + 10?
  • sample videos shared are face blurred. Will such blurring be removed in the final release? If not, do we know the effect of face blurring on pose tracking or mesh fitting? Will the authors share also 2D pose and SMPL or ANNY mesh extracted from the original videos without face blurring?
  • When reviewing clinical PD motion datasets (line 085) the authors can consider including other video and pose datasets such as PD4T and PD-GAM.
  • Please fix capitalization of your references. For example, Human3.6m should be Human3.6M and Weargait-pd should be WearGait-PD.
Reviewer_xHgfOpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

A small dataset filmed with six synchronised cameras, with triangulated 3D poses, UPDRS ratings for fist and gait, and 10 DBS patients recorded with stimulation OFF and ON. Three benchmark tasks. Main claims are that MPJPE doesn't track clinical feature error, off-the-shelf models fail on PD gait, and kinematics pick up DBS changes UPDRS misses.

Critical strengths
  • raw multi-view video with calibration and clinical ratings is interesting
  • evaluating clinical feature error and not only MPJPE is a good idea
  • paired DBS OFF/ON data doesn't exist publicly
  • the residualised target for response prediction is good practice
Critical weaknesses
  • re ground truth with triangulated keypoints, this shows 55 mm on gait but the lifters are 36 mm. How can you rank models within 2 mm here? Do perhaps like OpenCap did (PLoS Comput Biol 2023;19(10):e1011462) and include a small mocap subset
  • are you using the same 2D detector for both triangulation and as input for the lifter? If so, the in-domain lifter may simply learn your pipeline.
  • 140 mm for WHAM, PARE, and similar models on plain walking suggests a joint mapping or synchronization issue. Is this related to differences between SMPL and MediaPipe joints?
  • the multi-view upper bound is outperformed by monocular VideoPose3D on the fist task. noise? There are no CIs, and gait UPDRS 3 includes only three subjects.
  • you used 18 features with a sample size of 10 and applied the Wilcoxon test, but did not correct for multiple comparisons.
  • was OFF always recorded first? order effects on gait
  • since the same clinician rated body parts while watching the same videos, the correlation does not provide independent validation.
  • a responder prediction accuracy of 60% with 6 versus 4 patients does not constitute a meaningful task. Was there only one rater for UPDRS assessments?
Questions for authors

see above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.