acceptodds
Under review as a conference paper at ICLR 2027

OmniDIS: Dichotomous Image Segmentation with Detail-Reference Tokens

Abstract

High-fidelity dichotomous image segmentation requires both selecting the intended foreground and preserving its fine structure. Latent generative models offer strong visual priors, but compressed image representations can weaken boundary evidence, while a caption may match multiple objects. We present OmniDIS, which conditions image-to-mask flow on two complementary sources of evidence. Language–Point Dual-Prompting (LPDP) places semantic descriptions and labeled spatial cues in a shared context sequence, enabling text, point, and joint prompting with one model. Detail-Reference Token Injection (DRTI) converts multi-scale image residuals into learned reference tokens that condition the velocity field before decoding. This design lets target selection and detail cues interact during mask formation, without a separate refinement head. A frozen decoder transmits mask-supervision gradients to both modules, while low-rank adaptation updates the pretrained transformer. On DIS5K, OmniDIS-9B reaches and on DIS-VD with four Euler steps. It surpasses all recent state-of-the-art methods, including FlowDIS and LawDIS, while OmniDIS-4B remains competitive with a substantially smaller generative backbone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Dichotomous image segmentation, Flow matching, Generative model, Vision foundation model.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CPMDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces a high-quality dichotomous image segmentation (DIS) method with two complementary conditioning sources. Language-Point Dual Prompting (LPDP) uses language and point prompt guidance in order to specify the segmentation target. This provides additional clues to the model when the intended foreground target is ambiguous in the input image. The Detail-Reference Token Injection (DRTI) conditions the velocity predictor network on multi-scale image residuals in order to improve the preservation of fine structures in the segmentation masks. Experiments on DIS5K report higher segmentation scores than previous state-of-the-art methods.

Critical strengths
  • Integrating language and point dual prompting capability into high quality DIS is well-motivated, as in some cases the desired result can not be achieved without a prompt, or with only a language prompt.
Critical weaknesses
  • The motivation behind detail-reference token injection is not apparent. It seems that the image details would be lost after pooling the residuals to 8x8. Therefore, it’s unclear how the tokens derived from these 8x8 residual representations should help to preserve the segmentation details.

  • The writing quality of the paper is poor. The entire method section is hard or impossible to understand. It looks overly AI-generated without much human effort. For example, on line 225 it is not clear how each band is normalized and what the pooling operation is. In equation (8) I don’t understand what the channel-mixing operator A is, as it is never defined in the paper. In equation (9) I don’t understand what the vec is. In addition, the implementation details are mixed with the proposed method making it harder to understand the contribution of this work.

  • The ablation study in Table 4 is not clear. The paper does not explain the experiment settings for “No refinement”, “VAE enc. feats”, “DRTI feats.” and “DRTI (in latent)”. This makes it impossible to evaluate the true effectiveness of the proposed DRTI.

Questions for authors
  • What motivates using average pooling followed by bilinear upsampling in the multi-scale residual descriptor computation, rather than simple Gaussian smoothing?

  • How do the authors obtain the mask probability map from VAE-decoded results for the BCE calculation during the training?

  • It appears that 4 sampling steps were used for the quantitative and qualitative comparisons. Have the authors compared 1 step results?

  • Do you use the base or distilled variants of FLUX.2 [klein] for the experiments?

  • Citation formatting is inconsistent throughout the manuscript. Please follow the ICLR recommendation of using the parenthesis style where applicable.

Reviewer_aftXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OmniDIS addresses target ambiguity and the loss of fine structures in latent generative dichotomous image segmentation. Language–Point Dual-Prompting (LPDP) combines text descriptions with foreground/background-labeled point prompts to specify the intended target. Detail-Reference Token Injection (DRTI) converts multiscale RGB residuals into learned visual-reference tokens that provide additional detail cues. Both modules condition the image-to-mask flow before decoding, allowing target information and detail cues to interact during mask generation.

Critical strengths

The motivation for combining language and spatial prompts is clearly presented. The prompt-mode ablation supports their complementary roles, particularly in multi-object scenes.

Critical weaknesses
  • The ablations demonstrate that DRTI improves performance, but do not clearly establish whether high-frequency residuals offer an advantage over other image-derived reference representations.
  • Evaluation of spatial prompting is limited to a fixed protocol with two foreground and two background clicks sampled from ground-truth masks. Sensitivity to click count and placement remains insufficiently explored.
Questions for authors
  • How do multiscale residual descriptors compare with descriptors derived from RGB or low-pass images? Such a comparison would help establish the specific benefit of residual information.
  • How does segmentation performance change when more than two foreground clicks or more than two background clicks are provided? Please evaluate the effects of varying these counts independently.
  • How does the placement of foreground and background clicks affect segmentation quality? Please provide a quantitative sensitivity analysis and qualitative examples showing when adding point prompts improves or degrades performance relative to text-only prompting. It would be helpful to distinguish the effects of different valid click placements from those of inaccurate clicks.
  • What motivates channel-concatenating the clean image latent zIz_I with ztz_t and widening the input projection xembx_{\mathrm{emb}}, instead of supplying zIz_I through FLUX.2’s existing visual-reference interface? Have you compared these two conditioning strategies?
Reviewer_v6SfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes OmniDIS, a promptable dichotomous image segmentation (DIS) method built on FLUX.2 [klein] (4B / 9B). Following the image-to-mask flow formulation of FlowDIS, the model transports the image latent zIz_I to the mask latent zMz_M along zt=(1−t)zM+tzIz_t = (1-t)z_M + t z_I and predicts the velocity with a LoRA-adapted transformer. On top of this, the paper adds two conditioning modules. (1) LPDP appends point tokens (Fourier coordinate features + polarity embedding, with 4-axis RoPE identifiers on the packed latent grid) after the projected text tokens of the frozen Qwen3 encoder. (2) DRTI extracts three Laplacian-like RGB residual bands, pools them to an 8×88\times8 grid, and maps the resulting 16×8×816\times8\times8 descriptor into 256 reference tokens that enter FLUX.2's native reference-image pathway. Training also adds decoded BCE+Dice losses on the one-step endpoint estimate through a frozen VAE decoder. On DIS5K, OmniDIS-9B reports Fβω=0.949F_\beta^\omega = 0.949 and Sα=0.964S_\alpha = 0.964 on DIS-VD with 4 Euler steps, and claims to outperform FlowDIS, LawDIS and other recent methods.

Critical strengths
  1. The paper is generally clearly written and the method section is detailed. Tensor shapes, initialization, click sampling, and the gradient path through the frozen decoder are all described (Appendix A–D), which helps reproducibility.
  2. Having text, point, and joint prompting in one model is a practical idea for interactive DIS, and the multi-object split in Table 5 is a reasonable way to show where clicks help.
  3. The factorial ablation in Table 12 (reference tokens vs. auxiliary detail loss) is a good design. It is nice that the authors tried to separate the effect of the module from the effect of an extra loss.
  4. The 4B variant reaching performance close to FlowDIS with a smaller generative backbone, and the step/FPS analysis in Table 8, are useful for practitioners.
Critical weaknesses

W1. The main comparison in Table 1 is not done under matched inputs, so the SOTA claim is not well supported.

At evaluation time, OmniDIS receives a VLM-generated caption and four clicks sampled from the ground-truth mask (two foreground, two background; Sec. 4.1 and Appendix B.3). None of the methods in Table 1 receive these clicks. Most baselines are RGB-only, FlowDIS and LawDIS use text, and SAMA is evaluated with bounding-box prompts in its original paper. Appendix B.1 itself says the comparison is not an identical-input superiority claim, but the abstract and Sec. 4.2 still state that OmniDIS "surpasses all recent state-of-the-art methods".

The paper's own results show that the prompt mode has a large effect. In Table 5, adding clicks to text improves FβωF_\beta^\omega on the multi-object split from 0.913 to 0.938 (+2.5 points). For comparison, the margin over FlowDIS on DIS-VD in Table 1 is 1.1 points (0.949 vs. 0.938). So it is not clear how much of the improvement comes from the method and how much comes from the extra GT-derived clicks. This is made worse by two missing results. First, there is no RGB-only (no prompt) result for OmniDIS, even though most DIS baselines are RGB-only. Second, the prompt-mode study is only on DIS-VD, so a text-only comparison on DIS-TE1–4 is not possible. Also, OmniDIS uses 4 Euler steps, while the FlowDIS numbers in Table 1 are the 2-step results from the FlowDIS paper. This should at least be indicated in the table.

W2. Prompt sources differ across methods, and the click protocol is optimistic.

The text prompts used by OmniDIS and FlowDIS are produced differently. According to the FlowDIS paper, its prompts are generated by GPT-4V from images with the background blacked out (i.e., using the GT mask), and the detailed prompt is always used at test time. OmniDIS instead uses a Qwen3 VLM on the full image without the mask. The FlowDIS and SAMA numbers in Table 1 are identical to those in their original papers, so the text-guided methods were apparently not evaluated with a shared prompt set. The paper does not say which prompts the FlowDIS and LawDIS-R results correspond to. SAMA is also not a matched baseline in terms of training data: its paper trains on DIS-5K together with ThinObject-5K (and additional matting data), and evaluates on DIS5K with box prompts.

On the click side, Appendix B.3 says the cached clicks are reused "across evaluated methods that accept clicks", but as far as I can tell, no other method in Table 1 actually received clicks. SAMA accepts point prompts and could have been evaluated with the same four clicks, which would give at least one prompt-matched comparison. In addition, both training and evaluation clicks are sampled uniformly from the GT mask, and there is no evaluation with noisy, fewer, or more realistic clicks. The authors acknowledge this in the limitations, but since clicks are one of the two main contributions, I think this is a significant gap.

W3. Limited technical novelty of LPDP and DRTI.

LPDP. The text branch is simply the native text-conditioning path of FLUX.2 [klein] (frozen Qwen3 encoder and pretrained context projection), so the only new part is the point tokens. The point encoder, Fourier features of the click coordinates plus a learned fg/bg embedding, is conceptually the same as the SAM prompt encoder. The RoPE identifiers also follow the backbone's existing 4-axis RoPE convention, as the authors note in Sec. 2. LPDP is therefore a combination of existing components appended to the context sequence, rather than a new technique.

DRTI. Although the module is presented as a "detail reference" for recovering fine structure, the residual bands are pooled to an 8×88\times8 grid, so the whole descriptor is only 16×8×8=102416\times8\times8 = 1024 values. The authors also state that the reference slots do not correspond to image patches (Appendix C.1), and the tokens are injected through FLUX.2's existing reference-image pathway. This is closer to a compact global descriptor than to a detail reference. The ablations support this reading. A 4×44\times4 grid (256 values) already reaches Fβω=0.940F_\beta^\omega = 0.940 (Table 6), compared with 0.894 without reference tokens (Table 12), while going from 8×88\times8 to 16×1616\times16 adds only 0.001. If DRTI mainly provided spatial detail, performance should depend much more on grid resolution.

W4. Limited evaluation scope, and the name "Omni" seems overclaimed.

All experiments are on DIS5K only, with no cross-dataset evaluation, which the authors also acknowledge. The supported prompts are text and points only. With a single task, a single dataset, and two prompt types, the name "Omni" seems too strong. Either the name should be reconsidered, or zero-shot results on other high-resolution / thin-object benchmarks should be added to justify it.

Questions for authors
  1. Were the numbers for the baselines in Table 1 re-run by the authors or taken from the original papers? Which text prompts do the FlowDIS and LawDIS-R results correspond to? Could you evaluate OmniDIS in text-only mode on DIS-TE1–4 using the same prompts as FlowDIS, and report an RGB-only result?
  2. What exactly is the "Baseline" in Table 2? Does it use text prompts and LmaskL_\text{mask} ? Could you also report the full model trained without LmaskL_\text{mask} (flow loss only, as in FlowDIS)?
  3. How robust is the model to more realistic clicks, for example spatially perturbed clicks, fewer clicks (K=1K=1 or 22), or one click with a wrong label? Could SAMA be evaluated with the same cached clicks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.