HiSTAR: Hierarchical Semantic-aware Spatio-Temporal Decoding with Vision-Language Models for Radiology Scanpath Prediction

Radiology scanpath prediction aims to model the dynamic visual search behavior of radiologists during diagnostic reasoning. Radiological abnormalities typically exhibit low contrast and subtle visual manifestations, requiring integration of anatomical context with localized abnormal findings during visual search. Consequently, accurate scanpath prediction requires jointly modeling abnormality localization and the temporal evolution of visual attention, posing substantial challenges for existing methods. Recent medical vision-language models provide rich clinical context that complements purely visual representations. Motivated by these advances, we propose HiSTAR, a Hierarchical Semantic-aware Spatio-Temporal Decoding framework for radiology scanpath prediction. With HiSTAR, hierarchical semantic adaptation incorporates global and granularity clinical semantics to construct image-specific diagnostic search conditions and semantic-aware regional representations. Spatial-guided temporal decoding estimates a task-relevant spatial attention distribution as a prior for progressive trajectory generation. Finally, geometry-attention consistency further regularizes attention allocation and inter-fixation geometry to promote diagnostically relevant and spatially coherent scanpaths. HiSTAR achieves state-of-the-art performance on the GazeSearch benchmark, with a ScanMatch score of 0.3942. Downstream disease classification on SIIM-ACR and TB-MOUSE further demonstrates that HiSTAR-predicted gaze provides effective task-relevant information for medical image analysis. Code is available at https://anonymous.4open.science/r/HiSTAR.