acceptodds
Withdrawn from ICLR 2027

ExpertRoute: Distilling Radiologist Visual Search into Compact Evidence for Chest X-Ray Report Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

For chest X-ray report generation, visual context is useful only if it preserves the evidence needed for faithful diagnostic description. We introduce ExpertRoute, which distills radiologists’ visual-search behavior into an image-conditioned expert prior and combines it with independent representation-space semantic coverage under an exact visual-context budget. The expert prior is learned once from eye tracking and requires no gaze at inference. On a frozen LLaVA-Rad backbone, ExpertRoute reduces the visual context entering the LLM from 1,369 to 192 tokens (14.0%). Despite this reduction, it improves RadGraph and GREEN over full-context inference across three evaluation cohorts: MIMIC Validation, a locked MIMIC held-out cohort, and OpenI. On the locked held-out cohort, these RadGraph and GREEN gains are supported by patient-cluster bootstrap. Across the evaluated token budgets, ExpertRoute exhibits a non-monotonic response, with Micro-F1, Macro-F1, and RadGraph all reaching their best observed values at 192 tokens. At this operating point, the compact visual context also reduces per-sample generation latency by 16.2% relative to full-context inference. These results show that report quality depends on how visual evidence is composed, not simply on how much context is retained.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Vision-Language Models, Visual Token Routing, Expert Knowledge Distillation, Radiologist Gaze, Chest X-Ray Report Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tcm1OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ExpertRoute, a visual-token routing method for chest X-ray report generation. It trains a lightweight gaze scorer on radiologists’ eye-tracking data from REFLACX, then combines the predicted expert-priority regions with an independent semantic-coverage selector. The method retains exactly 192 of the original 1,369 visual tokens before LLM decoding, while keeping the vision encoder, projector, and report-generation backbone frozen. The experiments use MIMIC Validation, a locked MIMIC held-out cohort, and external OpenI data. At the 192-token setting, the method improves report-level RadGraph and GREEN scores over full-context inference across all three cohorts, with a reported 16.2% reduction in generation latency. However, finding-level CheXbert performance is not uniformly improved, and the approach does not reduce computation in the upstream vision encoder.

Critical strengths
  • Combines radiologist-derived visual priorities with semantic coverage, addressing the limitations of relying on either expert attention or model-derived selection alone.

  • Evaluates on a locked patient-level MIMIC cohort and an external OpenI dataset, with matched prompts and a frozen backbone. Includes recent token-reduction methods and controlled variants that help isolate the contribution of the proposed composition strategy. Reports paired patient-cluster bootstrap confidence intervals for key held-out comparisons. The RadGraph and GREEN gains over full-context inference have intervals excluding zero. Measures actual latency rather than relying only on token counts.

Critical weaknesses
  1. Evaluation relies primarily on automated metrics. Better RadGraph or GREEN scores do not establish that generated reports are clinically safer or more diagnostically accurate. Prospective radiologist assessment is absent.

  2. Full-context inference performs better on finding-level CheXbert F1 in the locked cohort. ExpertRoute therefore does not establish that compact routing preserves all clinically relevant evidence.

  3. ExpertRoute is not consistently the best compact-context method across metrics and datasets. For example, VisionZip has higher RadGraph and GREEN point estimates on MIMIC Validation, while VisPruner has stronger finding-level F1 on the locked cohort. Reducing LLM input tokens by 86% yields a 16.2% latency reduction. This is understandable because the vision encoder and projector still process all 1,369 patches, but the practical speedup is less substantial than the token reduction might suggest.

  4. The gaze scorer is trained on REFLACX, whose supervision is primarily based on frontal radiographs. The experiments do not establish transfer across different backbones or a broad range of clinical settings.

  5. The 192-token operating point needs careful interpretation. It is the best observed budget for the proposed policy among the tested values, not a universal optimum. The best routing strategy changes with the budget, and the paper does not establish how robust this operating point is to other datasets or generators.

Questions for authors
  1. Can the authors provide radiologist evaluation of report correctness, omissions, and potentially harmful errors to establish whether the automated-metric improvements translate into clinical benefit?
  2. Why does ExpertRoute underperform full-context inference on finding-level F1 in the locked cohort, and which findings are most affected?
  3. How much of the improvement comes from gaze supervision itself versus the independent semantic selector? Are the gains consistent across multiple random seeds and alternative gaze-scoring architectures?
  4. What are the separate runtime and memory costs of the vision encoder, projector, semantic selector, and LLM? How would earlier-stage routing or approximate semantic selection affect end-to-end performance?
  5. Does the method transfer to other vision-language backbones, hospitals, and radiographic views without retraining the gaze scorer?
  6. How sensitive are the results to the choice of 192 tokens, and can the operating point be selected using validation data without relying on the final evaluation cohorts?
Reviewer_cPZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper describes the ExpertRoute framework, which distills radiologist visual-search behaviour from eye-tracking, into an image-conditioned expert prior towards improving chest X-ray (CXR) report generation. This was demonstrated to improve CXR report generation on the RadGraph and GREEN metrics, despite significantly reducing visual context token count (which was also optimized by experiment) that is presented to the downstream LLM (under three patch selection policies). The major contribution is the evidence that radiologist gaze context is helpful for automated CXR input evidence filtering after suitable modelling, which moreover does not require human intervention at inference time.

Critical strengths
  • Presents a framework for the incorporation of expert gaze behaviour into aiding visual focus for LLM inputs
Critical weaknesses
  • Lack of comparison of the proposed human gaze-trained prior, against common saliency heatmaps from classification or other models
  • Minimal (or no) performance advantages over prior methods (Tables 1 and 2), even for multiple patch/token selection schemes
Questions for authors
  1. While the proposal of utilizing radiologist gaze data as visual supervision is relatively novel, the perhaps natural comparison of gaze heatmaps against saliency (e.g. Grad-CAM) heatmaps as generated by classifiers trained on the same CXR data, was not pursued. This has practical implications regarding on the value of collecting (expensive) gaze data, as opposed to heatmaps obtainable by automated means. If relevant, such a comparison may be attempted or discussed.

  2. From Section 3, Figure 2 and the results (Tables 1 and 2), patch selection policy appears to have a relatively minimal impact on performance, though the combination of expert-derived and non-overlapping tokens does make some intuitive sense. A random selection of patches (within each token budget) might therefore be considered as a further comparison, to quantify the value of patch/token selection policies.

  3. In Sections 3.2 and A.2, the gaze fixation data collection methodology may not have been sufficiently described. In particular, it is unclear as to the duration and density of REFLACX fixations, which does not appear fully characterized in the original REFLACX paper (Scientific Data 9, 350 [2022]) either. This might be clarified in the Appendix.

  4. Related to the above, Figure 3c in the REFLACX paper suggests that radiologist gaze tends to be diffuse for actual CXRs, and moreover may not necessarily meet the patch budget (up to 256 of 1369, for patches with non-zero responses) explored in this study. This might be discussed, with patch/anchor statistics presented since a gaze cap is implied (Section B.1).

  5. Moreover, it might be confirmed if gaze maps in Figure 1A and 4 are the actual fixation maps for the example CXR.

  6. In Section 4.1, it might be explicitly clarified as to whether the gaze model had possibly been trained on data used for ExpertRoute validation, since the REFLACX data had also been annotated on MIMIC-CXR.

Reviewer_5pFmOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ExpertRoute, a visual token selection framework for chest X-ray report generation. The method uses radiologists’ eye-tracking data to train a GazeScorer that predicts expert attention regions in the image, and combines it with an independent Semantic Coverage strategy to retain key visual information under a fixed token budget. Experiments based on a frozen LLaVA-Rad reduce the number of visual tokens fed into the LLM from 1,369 to 192, while achieving better RadGraph and GREEN results than the full-context baseline on the MIMIC and OpenI datasets.

Critical strengths
  1. Introducing radiologists’ eye-tracking behavior into visual token selection has reasonable clinical motivation and provides a meaningful research direction for visual information compression in medical VLMs.

  2. The experiments are relatively comprehensive, covering multiple token pruning baselines, different token budgets, independent test sets, external datasets, and inference efficiency evaluation.

  3. The paper provides architecture ablations of GazeScorer, overlap analysis of token selection, disease-level performance statistics, and uses Bootstrap to assess the statistical significance of some results.

Critical weaknesses

1. Insufficient validation of the learned gaze prior

The paper evaluates GazeScorer using metrics such as SIM and Top-192, but lacks reference baselines and upper bounds that reflect its actual prediction quality. It's better to add Center-prior (or Training-set Average Gaze) Top-192 Selection as a baseline, and Ground-truth Gaze Top-192 Selection as an oracle upper bound. These comparisons would help determine whether GazeScorer truly learns image-dependent expert visual search patterns, rather than merely a fixed spatial attention bias.

Although the paper shows some visualizations of gaze selection, it lacks systematic analysis under different disease conditions. It remains unclear whether GazeScorer can select corresponding key visual regions for different lesions. Providing qualitative or quantitative analysis of disease-stratified gaze selection will help explain the problem. For example, datasets with bounding-box annotations such as MS-CXR could be used to evaluate the coverage of lesioned regions by the selected patches; the spatial selection distribution similarity across different diseases could also be calculated. Such analyses would help verify whether the learned gaze prior has disease-related selection capability, rather than mainly relying on generic spatial distributions.

2. Potential disease-dependent selection bias

Table 6 shows that while ExpertRoute improves performance on some diseases, it also notably reduces Recall and F1 for certain diseases, especially Atelectasis, Lung Opacity, and Fracture. Considering that GazeScorer is trained on the relatively limited REFLACX dataset, the learned gaze prior may be affected by the disease distribution in the training data, thereby biasing toward certain diseases or anatomical regions. It's better to provide the disease distribution of the GazeScorer training set and analyze whether it is associated with the downstream Recall/F1 changes for each disease.

Questions for authors

Same as weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.