acceptodds
Withdrawn from ICLR 2027

HiSTAR: Hierarchical Semantic-aware Spatio-Temporal Decoding with Vision-Language Models for Radiology Scanpath Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Radiology scanpath prediction aims to model the dynamic visual search behavior of radiologists during diagnostic reasoning. Radiological abnormalities typically exhibit low contrast and subtle visual manifestations, requiring integration of anatomical context with localized abnormal findings during visual search. Consequently, accurate scanpath prediction requires jointly modeling abnormality localization and the temporal evolution of visual attention, posing substantial challenges for existing methods. Recent medical vision-language models provide rich clinical context that complements purely visual representations. Motivated by these advances, we propose HiSTAR, a Hierarchical Semantic-aware Spatio-Temporal Decoding framework for radiology scanpath prediction. With HiSTAR, hierarchical semantic adaptation incorporates global and granularity clinical semantics to construct image-specific diagnostic search conditions and semantic-aware regional representations. Spatial-guided temporal decoding estimates a task-relevant spatial attention distribution as a prior for progressive trajectory generation. Finally, geometry-attention consistency further regularizes attention allocation and inter-fixation geometry to promote diagnostically relevant and spatially coherent scanpaths. HiSTAR achieves state-of-the-art performance on the GazeSearch benchmark, with a ScanMatch score of 0.3942. Downstream disease classification on SIIM-ACR and TB-MOUSE further demonstrates that HiSTAR-predicted gaze provides effective task-relevant information for medical image analysis. Code is available at https://anonymous.4open.science/r/HiSTAR.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Radiology scanpath prediction, Vision language model, Hierarchical Semantic Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Koz2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HiSTAR, a framework for radiology scanpath prediction. It combines hierarchical clinical semantics from VLMs, spatial-guided temporal decoding, and geometry-attention consistency to generate scanpaths. Experiments on GazeSearch show improvements over existing methods, with additional downstream classification results.

Critical strengths
  • Interesting Problem: Predicting radiologists' scanpaths is a meaningful task with potential clinical applications.
  • Effective Framework: The proposed method combines clinical semantics with spatial and temporal modeling and achieves strong performance.
  • Detailed Experiments: The paper provides comparisons with existing methods, ablation studies, and downstream classification experiments.
Critical weaknesses
  • Unclear Novelty: The main components, including VLM-based semantic features, spatial attention, temporal decoding, and geometric constraints, are related to existing approaches. The paper should better clarify its technical novelty and new insights beyond combining these components.
  • Limited Baseline Comparison: HiSTAR uses a strong external VLM to generate detailed clinical descriptions, while existing baselines do not use the same semantic information. It is unclear how much improvement comes from the proposed architecture rather than stronger VLM features.
  • Limited Generalization: The main radiology scanpath evaluation is conducted only on GazeSearch, which focuses on target-present chest X-ray search. The performance on other medical datasets and target-absent cases remains unclear.
  • VLM Dependency and Cost: The method relies on VLM-generated global and region-level descriptions. The impact of different VLMs, incorrect clinical descriptions, and the additional inference cost is not sufficiently evaluated.
Questions for authors
  1. Can the authors compare against baselines using the same VLM-generated semantic features?
  2. How robust is the method to inaccurate VLM descriptions or different VLM backbones?
  3. Can the authors provide further evaluation on independent medical scanpath datasets or target-absent cases?
  4. What is the total inference cost of generating the global and region-level clinical descriptions?

I am willing to adjust my score according to the rebuttal.

Reviewer_bpoMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper predicts radiologists' scanpaths conditioned on a chest radiograph and a diagnostic search target, including fixation locations, durations, and termination. HiSTAR uses global and regional descriptions from a vision–language model to construct task conditions and regional representations. It predicts a history-independent spatial attention map, then decodes a trajectory using fixation history, with additional attention-allocation and saccadic-geometry objectives. Evaluation covers GazeSearch, disease classification using attention maps derived from predicted scanpaths, and transfer to natural-image search.

Critical strengths
  1. The main benchmark on scanpath prediction shows clear improvements.
  2. The integration of global semantics, regional semantics, spatial priors, and trajectory geometry has a coherent motivation. Related component ablations are also provided in Table 3-4.
Critical weaknesses
  1. The previous work LogitGaze-Med already integrates patch-level vision–language semantics into medical scanpath prediction. This paper has distinct design choices and introduces richer global and regional descriptions, but it is more like a incremental work built on LogitGaze-Med.
  2. The paper entangles architectural improvements with additional semantic resources. The ablations support component utility within this system, but do not adequately establish the remaining gain when strong baselines receive the same semantic inputs, backbone, and training budget. Resource-matched comparisons and end-to-end costs analyze is needed.
  3. Table 2 includes ChestSearch and human gaze, but not classification using LogitGaze-Med predictions.It therefore does not establish whether the scanpath advantage translates into diagnostic gains over the closest prior work.
Questions for authors

With identical global/regional semantics, backbone, and training budget, what advantage remains over LogitGaze-Med or a simpler decoder?

Reviewer_aJtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents HiSTAR, a method for predicting where radiologists look when examining chest X-rays. It uses VLM-generated clinical descriptions, spatial attention, and a Transformer to predict eye movements. HiSTAR is evaluated primarily on GazeSearch, where it reports improvements over the baselines. The paper also assesses the usefulness of predicted attention for classification on SIIM-ACR and TB-MOUSE and reports cross-domain evaluation on COCO-Search18.

Critical strengths

The formulation into semantic adaptation, spatially guided temporal prediction, and consistency regularization is understandable and well aligned with the stated radiology scanpath prediction task.

HiSTAR achieves the best reported performance among the baseline methods in Table 1, with particularly notable improvements in duration-aware ScanMatch and MultiMatch duration.

Meaningful ablation studies. The paper provides extensive ablations examining loss components, global versus local semantics, and spatial guidance, along with an analysis of sensitivity to loss weights. It also evaluates cross-domain generalization.

Critical weaknesses

The comparison with LogitGaze-Med needs clarification. The paper states that LogitGaze-Med was reproduced under the same protocol. However, the reproduced ScanMatch scores are substantially lower than those reported by the original authors, while the scores for shared baselines (GazeFormer, HAT, ChestSearch, and LogitGaze) are relatively similar across the two reports. The paper does not clearly explain this larger discrepancy, making it difficult to assess the claimed improvement over LogitGaze-Med.

Limited methodological validation. Quantitative evidence linking the three research gaps to specific baseline failures remains limited. The experiments do not clearly distinguish improvements from richer VLM-generated clinical descriptions versus the proposed architecture.

The cost and reproducibility of semantic preprocessing need clearer reporting. Generating global descriptions and processing 49 regions per image introduces substantial potential VLM overhead. The paper does not report preprocessing cost, inference latency, or computational comparisons with existing methods. Testing another VLM would be a useful robustness check; evaluating alternative resolutions would clarify sensitivity to the choice of 7x7 grid.

Questions for authors

Which LogitGaze-Med variant was reproduced, and what training settings and scanpath-generation procedures were used? In particular, why is the discrepancy between the reproduced LogitGaze-Med results and the original authors’ reported results substantially larger than the corresponding discrepancies for other shared baselines?

How are the 18 COCO-Search18 targets represented when the trained task-embedding vocabulary contains only 13 radiological findings, and is any additional fitting performed?

How stable are the main GazeSearch gains across training seeds, and what uncertainty estimates can be provided for the principal comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.