SpineDiag: A Multimodal Spine Scoliosis Benchmark for Radiation-Free Diagnosis

Adolescent idiopathic scoliosis (AIS) is a common spinal deformity. However, large-scale diagnosis and repeated follow-up are limited by the radiation exposure of full-spine X-ray. To this end, we build SpineDiag, a paired multimodal clinical AIS dataset with 155 subjects and radiation-free modalities (visual observations, depth maps, and point clouds, annotated keypoints, and physician-verified diagnosis descriptions). To build a unified spine representation, we first design the 2D spinal keypoint protocol from medical expertise, then lift this 2D anatomical representation with depth to reconstruct a 3D kinematic spine model in simulation. We further propose a benchmark suite with four tasks: (1) visual spine perception, (2) visual diagnosis with video only, (3) visual diagnosis with video and 2D keypoints, and (4) visual diagnosis with videos and depth map. The results are assessed with keypoint and geometric metrics, a clinically Visual Question Answering (VQA) score, and reported LLM verification diagnosis score. We train and evaluate pose estimators and vision-language models (VLMs) on these tasks. The experimental results expose open problems of current models in radiation-free diagnosis, which needs further investigation and more robust models.