acceptodds
Withdrawn from ICLR 2027

XFlow: A Workflow Model for Instruction-Guided Lesion Segmentation in Chest X-rays

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing text-guided segmentation models in the medical domain cover only a narrow set of anatomical structures and lesions in chest X-rays (CXRs), and most of them assume that the queried target is always present in the image. Instruction-guided lesion segmentation (ILS) was introduced to overcome these limitations by segmenting diverse lesion types from simple user instructions while also recognizing when the queried lesion is absent, and ROSALIA was proposed as the first model for this task. However, the masks produced by ROSALIA remain of limited quality, often carrying scattered noise. Moreover, ROSALIA predicts the mask in a single shot, which differs fundamentally from how radiologists perceive and delineate lesions in practice. A radiologist first surveys the entire thorax, then localizes the approximate region of abnormality, and only then refines the lesion contour. Motivated by this coarse-to-fine, multi-level perception process, we present XFlow, a workflow model for ILS that combines box-based localization with multi-turn point refinement. XFlow detects the lungs, decides whether the queried finding is present in each of them, and prompts a fine-tuned SAM with the lesion box for an initial mask. It then corrects that mask through point prompts until its boundary follows the lesion, leaving every intermediate decision visible. Our experiments show that XFlow achieves the best segmentation quality on both internal and external evaluation. Notably, it surpasses ROSALIA in segmentation quality even when the two are trained on the same lesion annotations.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Chest X-ray, Lesion Segmentation, Vision Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VuXJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a workflow model for instruction guided lesion segmentation in chest x-rays. It uses a two-stage pipeline built on MedGemma to generate box and coordinate prompts that are used by a separate SAM3 model to produce segmentations. Supervised fine-tuning and GRPO are used to train each policy. The model is evaluated on an internal test set and an external test set from a different hospital. There is an additional analysis of fragmentation in the predicted masks, and a reader study with medical experts.

Critical strengths

The methodology is strong. The authors retrain the baseline ROSALIA on the same training cases. The evaluations are also strong and include an external dataset and a reader study. The analysis of mask fragmentation shows a clear advantage of XFlow over ROSALIA.

Critical weaknesses

One of the main claims of the paper is that the multi-turn workflow is beneficial to segmentation performance. The ablation study, however, suggests that stage two doesn't meaningfully improve segmentation performance and that nearly all of the improvement over ROSALIA is because of the SFT and RL of stage 1. Stage two shows only minimal improvements on the internal CheXpercept dataset and almost no benefit on the external CheXlocalize.

Questions for authors

To address the small performance margins in stage 2 ablations, would the authors consider including variation across seeds, confidence intervals, or some other analysis to help readers judge the significance of these slight differences in performance?

Would the authors consider including a baseline on a simple multi-channel segmentation model (e.g. nnU-net) trained on the same dataset to demonstrate the benefit of instruction-guided segmentation?

Reviewer_a4BVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper talked about XFlow, which is a workflow using vision language models with 2 stages. The essence is an instruction guided lesion segmentation model in chest xrays. It is first given an instruction to name one of the seven findings and a scope. Then in stage 1, a MedGemma 4b model locates the region of interest, and output bounding boxes. The boxes would be used as box prompts for a SAM3 finetuned on chest xrays to produce segmentation masks. In stage 2, another model would give up to 4 positive or negative clicks to the masks. After all that, they trained it with reinforcement learning for both stages, stage 1 the reward is box overlap, and stage 2 the reward is overlaps gained by the clicks. They then tested the workflow on an expert curated subset of CheXpercept and another dataset called CheXlocalize. XFlow is able to beat all the baselines, as well as ROSALIA retrained on the same data. Also they have reader studies showing that physicians would prefer XFlow’s masks.

Critical strengths
  1. The training and comparison is fair, they trained the competitors on the same data. With the fair comparison, XFlow still beats all competitors. I would deem this performance gain legit.
  2. The reader study is blinded and randomized. This is very good practice when it comes to medical imaging studies.
  3. The author also honestly reported real failures that overlap scores miss. And admitted that Stage 2 does not add much performance.
Critical weaknesses
  1. The main claim that ‘radiologist-style workflow’ causes the gains is not well supported. From the paper table 2, we can see that stage 1 alone already accounts for 57% of the internal gain, and 70% of external gain. The multi-step clicks only add 0.01 internally and 0.003 externally. Given that XFlow changes multiple things at once, including a different language model, a stronger segmentation model, and extra lung box supervision, it is hard to draw the conclusion that it is the ‘radiologist-style workflow’ that boosted the performance.
  2. The ground truth for internal could be biased, if they are the masks described in the CheXpercept paper. The paper says that the masks were regenerated with ROSALIA, and reselected by experts. This process could potentially remove the hard cases, and leave easy cases for the internal dataset, especially when XFlow itself is tuned to ROSALIA’s style. (this will give XFlow advantage over other baselines, but of course this also means that the bias would give ROSALIA more advantage, so XFlow beating ROSALIA is still a legit claim)The reader study also might have a similar problem, where the readers were the curators of CheXpercept, so essentially they are rating masks that they previously approved.
  3. There is potentially data overlap between the training and testing split. When they are doing reinforcement learning, they clearly stated that they are splitting the patients. But for training and testing split, they said its based on xrays, not by patient. Not sure if this is a typo or a mistake.
Questions for authors
  1. If we keep the language model and segmentation model the same, how much performance would the workflow still add? Or, what if we do a one shot model using the same MedGemma-1.5-4b and the fine-tuned SAM3 training on the same data? It would be great if we can see how much performance is because of the workflow, and how much is because of stronger models.
  2. Does the internal test have patient overlap? Are any of the test image part of MedGemma’s training set?
Reviewer_HqwvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes XFLOW, a multi-stage workflow model designed for instruction-guided lesion segmentation in chest X-rays. The pipeline aims to mimic clinical radiologist behavior through a multi-step inference process using RL and SFT training: first, it localizes the lung region within the X-ray; second, it verifies whether a queried target object/lesion is present; third, it employs a prompt-based Segment Anything Model (SAM) to generate an initial segmentation mask; and finally, it iteratively refines the mask via point prompting until the bounding box accurately encompasses the lesion.

Critical strengths

S1. Clear Writing & Organization: The manuscript is well-structured, clearly written, and easy to read and navigate.

S2. Intuitive Clinical Motivation: Structuring interactive segmentation as an agentic workflow that mimics a radiologist's step-by-step diagnostic reasoning is an intuitive concept.

Critical weaknesses

Major:

W1. Highly Narrow Domain Scope & Limited Experimental Validation: The proposed method is heavily customized for chest X-ray lesion analysis, and the evaluation is conducted on a narrow set of conditions without demonstrating generalizability to broader medical image segmentation tasks.

W2. Lack of Robustness Across Diverse Datasets: The experimental evaluation lacks robustness. The framework is not extensively evaluated across multiple benchmark chest X-ray datasets from varying clinical centers, imaging vendors, or patient demographics.

W3. Insufficient Theoretical Grounding & Applicability Bounds: There is a notable lack of in-depth theoretical analysis justifying why the multi-step heuristic workflow functions effectively. The authors do not establish formal operational boundaries, failure modes, or guidelines on where this methodology degrades or should not be applied.

W4. Absence of Baseline Comparisons with RL/Instruction-Guided SAM Methods: The paper lacks comparative experiments against relevant recent paradigms that combine SAM with reinforcement learning (RL) or instruction tuning for interactive segmentation. A comparison with state-of-the-art interactive agent/prompting methods is missing.

Questions for authors

Q1. Generalizability across domains: Does the proposed workflow extend to other anatomical regions (e.g., CT/MRI brain or abdominal segmentation), or is it strictly constrained to 2D chest X-rays?

Q2. Cross-dataset evaluation: How does XFLOW perform when evaluated on out-of-distribution clinical chest X-ray datasets?

Q3. Computational Overhead: What is the computational latency and prompt overhead during inference compared to single-pass prompt-able segmentation models?

Suggestions for improvement:

  1. Deep Clinical Validation: If the paper's scope is strictly targeted at chest radiology, conduct a thorough empirical study across at least 5 diverse datasets. Include statistical confidence intervals, model uncertainty metrics, rigorous failure/edge-case analyses, and a clear theoretical framework.

  2. General Segmentation Framing & RL Baseline Comparison: If positioning the contribution as a generalizable instruction-guided segmentation algorithm, test on diverse multimodal datasets outside chest X-rays. Furthermore, benchmarking against recent RL-based alignment and instruction-tuned vision-language models (VLMs) is essential.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.