acceptodds
Under review as a conference paper at ICLR 2027

X-Chat: Any Perception via Multi-Modal Dialogue

Abstract

Multimodal large language models (MLLMs) take a perception request as a turn of dialogue, whether it names a category, describes an object, asks a question, or points at a region. Their outputs are not unified, however: segmentation MLLMs return masks alone, and MLLMs that report boxes write them as text or decode them apart from masks, each covering part of the task spectrum. We present X-Chat, an MLLM that takes multimodal dialogue as its perception interface and covers the segmentation and detection forms of seven task families (generic, open-vocabulary, referring, reasoning, grounded conversation generation, interactive, and visual grounded), fourteen tasks in all. Its language model turns the instruction into condition embeddings that drive one perceptor decoder. Each object query of that decoder regresses a box in parallel with its mask, and a geometric consistency loss couples box and mask. A detection task therefore differs from its segmentation counterpart only in which prediction is returned, and box-only data enter joint training directly. Because the existing interactive and visual grounded benchmarks score masks only, we build and release LVIS-VGD and LVIS-INT for the detection form of these two families. With one set of weights, X-Chat reports results on all fourteen tasks and is compared task by task with X-SAM and with published specialist models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Modal, Detection, Segmentation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YaLNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

X-Chat unifies detection and segmentation across seven task families through a dialogue-conditioned perceptor. Shared object queries predict boxes and masks, coupled by a geometric consistency loss, while box-only data support joint training. The paper also introduces LVIS-VGD and LVIS-INT detection evaluations and reports fourteen tasks with one checkpoint.

Critical strengths
  1. This paper is clear and easy to follow.

  2. Figures and tables are helpful for understanding the paper.

  3. Supporting all 14 tasks with one ckpt is useful integration result, and the shared decoder directly addresses the fragmented output interface.

Critical weaknesses
  1. The paper does not clearly show the benefits of training these tasks in one model over using an existing MLLM with perception tools. A direct comparison with such a system is missing. For example, the MLLM could interpret user instructions and call SAM 3 for detection and segmentation. The experiments do not establish better accuracy or inference efficiency over this alternative. This is a major concern because unification is the paper’s main contribution.

  2. The comparison with X-SAM changes both the LLM and the perceptor. Shared box–mask queries also come from prior work (§2). Therefore, Table 5 does not isolate the benefit of joint detection–segmentation training. A comparison using the same backbone is needed to establish this benefit. This limits the evidence for the paper’s central contribution.

Questions for authors
  1. On the same point-prompt split, how does direct box prediction compare with boxes extracted from predicted masks?

  2. Does the interactive-segmentation drop persist in a matched-backbone comparison with and without detection training?

  3. Could the authors report the inference latency for representative detection and segmentation requests using the shared model?

2 comments

Thank you for noting the value of covering all 14 tasks with one checkpoint and a shared decoder.

Major concern & Q3. Comparison with an MLLM + SAM 3 pipeline and inference latency

We built a baseline pairing Qwen3-VL-4B-Instruct with SAM 3. For referring expressions, Qwen3-VL-4B-Instruct rewrites the user instruction into a target noun phrase for SAM 3, and we take SAM 3's top-scoring prediction. For generic detection, instance segmentation, and interactive prompts, inputs go directly to SAM 3 without an LLM call. Both systems were timed on the same single Ascend 910B2C device with batch size 1 and warm weights in memory:

1. Accuracy:

Benchmark and metricX-ChatQwen3-VL-4B + SAM 3Difference
RefCOCO val detection [email protected] (10,268 exprs)94.3972.35 (raw: 67.54)+22.04
RefCOCO val segmentation cIoU (10,268 exprs)84.6157.68 (raw: 52.74)+26.93
COCO generic detection AP / AP50 (1,000-image subset)56.05 / 74.3156.14 / 72.78−0.09 / +1.53
COCO-Interactive point segmentation cIoU (single click)52.4351.10+1.33

2. End-to-end latency and peak memory (batch size 1, 25 requests × 3 repeats per task):

TaskX-Chat p50 / p95 (s)Qwen3-VL + SAM 3 p50 / p95 (s)Note
COCO generic detection (80 classes)0.641 / 0.7232.790 / 2.8144.36× faster (p50)
COCO instance segmentation (80 classes)0.839 / 0.8972.815 / 2.8423.36× faster (p50)
RefCOCO referring detection0.547 / 0.5690.431 / 0.516Both under 1s
LVIS interactive point detection0.567 / 0.6190.138 / 0.147Both under 1s
Peak device memory (HBM)14.78 GB20.09 GB26.4% less memory

Comparing the two systems shows where training a single model pays off:

  • On RefCOCO val, X-Chat is higher by +22.04 [email protected] and +26.93 cIoU (and feeding the raw expression directly to SAM 3 scores 67.54 [email protected] and 52.74 cIoU). Turning a relational or spatial expression into a short text prompt for SAM 3 loses visual context that X-Chat keeps in its condition embeddings.
  • On 80-class COCO detection and segmentation, X-Chat is 4.36× and 3.36× faster at similar detection AP (56.05 vs. 56.14, and +1.53 AP50) because it decodes all categories in one pass, whereas SAM 3 has to run 80 category prompts one by one even when reusing image features.
  • Keeping one model in memory instead of Qwen3-VL plus SAM 3 reduces peak HBM from 20.09 GB to 14.78 GB. SAM 3 is stronger when given box prompts or multi-click sequences on interactive segmentation, while X-Chat is competitive on single-click cIoU (+1.33) and much faster on multi-category queries. We will add this comparison to the paper.

Q1. Direct box prediction vs. boxes derived from masks on the same point split

On all 35,991 point requests of LVIS-INT val, using the same X-Chat checkpoint, forward pass, and selected query, direct box regression outperforms taking the bounding box of the predicted mask:

Readout on LVIS-INT point (35,991 requests)Box mIoU[email protected][email protected]
Mask-derived box32.5730.6015.60
Direct box41.8438.1918.44
Gain (Direct vs. Mask-derived)+9.28 [8.79, 9.76]+7.58 [6.81, 8.40]+2.84 [2.14, 3.49]

Repeating evaluation with three inference seeds gives +9.270 ± 0.006 box mIoU and +7.611 ± 0.055 [email protected]. On the 29,889 requests with nonempty masks, direct boxes still lead by +4.78 box mIoU because fragmented or noisy masks distort the enclosing box. The same comparison on fixed weights also favors direct boxes on LVIS-VGD point (+4.39 AP, +5.34 AP75, +5.57 small object AP), LVIS minival (+3.50 AP), ODinW-13 (+2.05 AP), and COCO (+0.97 AP).

Q2. Backbone alignment and interactive segmentation under joint training

We agree that Table 5 compares complete systems with different backbones rather than isolating joint detection and segmentation training, as noted in Section 4.4. Retraining an X-SAM baseline from scratch on Qwen3-VL-4B + DINOv3-L with an identical hidden-state normalization interface was not feasible within the rebuttal period. Instead, our fixed-backbone comparisons in Q1 (direct box vs. mask-derived box on the same checkpoint) and Table 3 (where adding box-only Objects365 data improves ADE20K segmentation by +2.80 PQ, +3.99 mIoU, and +6.95 mask AP on the same backbone) test both parts of our design under controlled conditions. We will revise Section 4.4 to separate the 14-task system comparison in Table 5 from these controlled ablations, and note the interactive segmentation gap to specialist models as a trade-off of training one model for all 14 tasks.

Thank you for suggesting the Qwen3-VL + SAM 3 baseline. We implemented the pipeline you proposed and found that X-Chat gained 22.04 points in [email protected] on RefCOCO val. On 80-class COCO detection, median latency dropped from 2.79 s to 0.64 s at comparable AP. We’d be interested to hear how this comparison informs your assessment of the unified model’s practical value.

Reviewer_uLEJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents X-Chat, a multimodal model for detection and segmentation with a single set of weights. Driven by a language-conditioned perception module, the model leverages shared object queries to predict both boxes and masks simultaneously, and introduces a geometric consistency loss between the two predictions. In addition, this work constructs LVIS-VGD and LVIS-INT for visually prompted detection. My core concern is that the proposed method tends to be engineering-oriented, and the existing experiments are insufficient to verify its effectiveness.

Critical strengths

1.This framework achieves competitive results on multiple benchmarks. It unifies language-prompted and visual-prompt-driven perception tasks under a single interface, demonstrating promising practical application value. 2.In the ablation study, the core geometric consistency loss yields a 1.5 point improvement in b-box AP, providing preliminary evidence for its effectiveness.

Critical weaknesses

1.Shared box and mask prediction, as well as joint training using box-only annotated data, have been explored in prior works such as MaskDINO, MVP-LM and OpenSeeD. The geometric consistency loss is therefore a key source of methodological novelty, yet the ablation studies do not sufficiently validate it. 2. Under the box-prompt setting of LVIS-INT, simply copying the input prompt box yields an [email protected] of 98.5. The experiments has limitations for demonstrating newly acquired perceptual capabilities.

Questions for authors
  1. For the final configuration of Qwen3-VL-4B+DINOv3-L, under exactly the same data and training budget, how would removing the geometric consistency loss affect detection, segmentation, and consistency (box and mask)?
  2. What is the performance gap on compatible detection tasks when converting predicted masks from X-SAM or a segmentation model with matched configurations directly into bounding boxes? 3.What specific failure cases are addressed by the newly added direct box prediction compared with this low-cost baseline?
  3. The box-prompt copying in LVIS-INT reaches 98.5%. What new capability does this protocol intend to measure? Could the authors provide a prompt perturbation protocol with practical discriminative power, as well as a fair comparison between point prompts and VGD?
  4. Can Table 3 control for instruction format, the number of training updates per sample, and class folding strategy to verify the semantic effect of class names?
2 comments

Thank you for the detailed suggestions. Below are the results for all four questions.

Q1. Geometric loss in the final configuration

Same data and update budget (Qwen3-VL-4B + DINOv3-L), initialized from the matching DINOv3-L pretraining checkpoints:

Seed (w/o → with)Box APBox AP75Mask APMask APsBox & mask IoU ↑Mask outside box ↓
102456.13 → 56.0561.06 → 61.1151.64 → 51.6132.09 → 32.520.751 → 0.7666.40% → 4.91% (−23.3%)
102556.36 → 56.3361.32 → 61.3952.06 → 51.9432.76 → 32.940.751 → 0.7696.52% → 4.82% (−26.1%)

In DINOv3-L pretraining, the loss raises box AP in both seeds (+0.34, +0.91 AP; +0.50, +1.22 APs). At the final endpoint, overall box/mask AP stay on par, small object mask AP rises (+0.43, +0.18), and mask spill outside the box drops by 23.3% and 26.1% with no added parameters or latency. We will scope the +1.5 AP to Table 4 (DINOv3-S+) and add these DINOv3-L results. Section 3.2 cites MaskDINO for parallel heads; we will expand Section 2 to contrast X-Chat with MaskDINO, MVP-LM, and OpenSeeD (one dialogue model covering all 14 tasks, plus LVIS-VGD and LVIS-INT).

Q2. Mask-to-box baseline and failure cases

Same checkpoint, forward pass, and selected queries, comparing direct boxes against tight boxes of X-Chat's predicted masks:

BenchmarkMask boxDirect boxΔ [95% CI]Δ AP75 / APs
COCO val2017 (AP)54.8655.83+0.97 [0.76, 1.18]+1.15 / +1.64
LVIS minival (AP)40.6544.15+3.50 [3.11, 3.89]+4.19 / +4.33
ODinW-13 mean (AP)50.4852.54+2.05 [1.49, 2.53]/
LVIS-VGD point (AP)41.9546.33+4.39 [3.86, 4.75]+5.34 / +5.57
LVIS-INT point ([email protected])30.6038.19+7.58 [6.81, 8.40]/
LVIS-INT point (mIoU)32.5741.84+9.28 [8.79, 9.76]/

Gains hold on LVIS rare/common/frequent (+2.89/+3.28/+3.80 AP), 11/13 ODinW sets (Pothole +6.91, Raccoon +5.91 AP), and all 8 RefCOCO/+/g splits (+0.20 to +0.41 mIoU).

Failure cases (35,991 LVIS-INT point requests): (1) Empty masks (16.95%): on small/low-contrast objects, mask logits fall below threshold (0 IoU), while the direct box reaches 31.31 mIoU (21.60% [email protected]). (2) Fragmented masks (12.16%): disconnected pieces miss part of the target; direct boxes lead by +12.64 mIoU (+16.64 [email protected]). (3) Outlier pixels (24.85%): stray foreground pixels enlarge the mask box; direct boxes gain +3.96 mIoU (+3.76 [email protected]). On the 29,889 nonempty-mask requests, direct boxes still gain +4.78 mIoU (+4.72 [email protected]), and excluding empty masks on COCO/LVIS-VGD keeps the AP gains unchanged (+0.96, +4.32 AP).

Q3. Box prompt perturbation and point prompts vs. VGD

Copying unperturbed ground-truth boxes scores 98.5% [email protected]. To test box correction, we perturbed the input boxes on 2,048 LVIS-INT requests:

PerturbationCopy mIoUX-Chat mIoUΔ mIoUCopy [email protected]X-Chat [email protected]Δ [email protected]
20% shift60.3671.52+11.152.3953.22+50.83
Scale ×0.862.1170.59+8.485.3750.39+45.02
Scale ×1.270.8573.75+2.8937.2161.08+23.88

X-Chat refines noisy boxes toward object boundaries (+2.89 to +11.15 mIoU, +23.88 to +50.83 [email protected] over copying; 93% to 97% of perturbed boxes still have IoU ≥ 0.5 with ground truth, making [email protected] insensitive).

Point prompts vs. VGD (35,991 aligned instances): Direct boxes beat mask boxes on both LVIS-INT (41.84 vs. 32.57 mIoU, +9.28; 38.19 [email protected]) and LVIS-VGD (51.20 vs. 43.61 mIoU, +7.59; 51.12 [email protected]). VGD is higher overall because its prompt gives point exemplars for image categories and predicts category sets, whereas INT gives one point for one target.

Q4. Controlling instruction format, updates, and class folding in Table 3

Replacing all names with "object" folds 365 classes into one. Our permutation control on full Objects365 permutes all 365 names (no class mapped to itself) with the same instruction format, 365-class structure, boxes, and updates per sample:

Mixture / category namesADE PQADE mIoUADE APO365-only PQ (Lenient 11)O365-only PQ (Strict 20)
COCO seg only18.3221.1711.704.467.26
+ O365 (all "object")15.3317.2210.910.001.78
+ O365 (permuted)15.7418.5211.230.250.19
+ O365 (correct)21.1225.1618.6631.4229.60
Correct − Permuted+5.38+6.64+7.43+31.16+29.41

With format, schedule, and 365-class structure matched, correct names beat permuted names by +5.38 PQ, +6.64 mIoU, and +7.43 mask AP (31.42 vs. 0.25 lenient PQ, 29.60 vs. 0.19 strict PQ on ADE20K classes unique to Objects365).

Thank you for asking about the consistency loss in the final configuration. The clearest effect in our two-seed ablation was improved box–mask agreement. With the same data and training budget, the loss reduced the fraction of mask pixels outside the predicted box by 23–26%, while overall box and mask AP stayed comparable. Does this answer your question about what the loss contributes in the final model?

Reviewer_ReqtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

X-Chat is multimodal LLM reads the image and a conversational instruction whose phrases carry category names, referring expressions, reasoning questions, or visual prompts (point, scribble, box, or mask) pooled by the region sampler, and returns either masks with a single set of weights. Its LLM based on Qwen3-VL-4b produces phras and <PCT> token which drive DINOv3-L + MaskDINO perceptor, whose object queries each predict a mask and regress a box in parallel. The requested output form selects which of the two predictions is returned. During training, the geometric consistency loss couples each box with SoftBox of its mask. the paper reports results on all 14 tasks, compares with X-SAM and published specialists, and introduces LVIS-VGD and LVIS-INT for the detection forms of the visual grounded segmentation and interactive families.

Critical strengths
  • real 14-task coverage with one set of weights. the model shows results on all seven families in both masks and box form with strong absolute numbers (COCO 55.8, REC 94.4, LVIS 44.1, COCO PQ 57.4 and two benchmarks. this is a substantial system contribution.

  • The benchmark construction is careful and the paper is transparent about its own confounds: LVIS-VGD/INT use LVIS MiniVal from COCO val2017 to avoid training overlap, exclude non-exhaustively annotated instances, and hold out rare categories, and the paper states that the X-SAM comparison does not isolate the detection form, that LVIS is not zero-shot, and that the prompt box alone scores 98.5. This transparency and the benchmark itself are valuable to the community, and they make the paper’s claims auditable.

Critical weaknesses
  • headline comparison against X-SAM is confounded by the backbone. X-Chat uses Qwen3-VL-4B + DINOv3-L while X-SAM uses a different LLM and perceptor, and the paper itself says the comparison “does not isolate the effect of adding the detection form.” in line 475-476. since the claim that the unified design beats X-SAM is the main scientific claim, current evidence cannot distinguish the design from a stronger backbone. A backbone-matched comparison or capacity-matched comparision X-SAM’s formulation retrained with X-Chat’s LLM/perceptor, or X-Chat retrained with X-SAM’s backbone would support it.

  • The geometric-consistency gain is single-seed and non-converged. The +1.5 box AP is measured at 12 epochs when, by the authors' own statement, "all arms were still improving" and box AP was rising by more than the spread between arms. that is not evidence that the loss helps

Questions for authors
  1. can you provide backbone matched comparison ?
  2. does +1.5 box AP from geometric consistency loss survive training to convergence with more seeds ?
2 comments

Thank you for your positive assessment of the 14-task evaluation and the LVIS-VGD and LVIS-INT benchmarks.

Q1. Backbone-matched comparison

Within the same X-Chat checkpoint and forward pass, we compare the box regressed by each selected query against the tight bounding box of that query's predicted mask, keeping query scores and retained predictions fixed:

BenchmarkMask-derived box APDirect box APΔ AP [95% CI]Δ AP75Δ Small object AP
COCO val201754.8655.83+0.97 [0.76, 1.18]+1.15+1.64
LVIS minival40.6544.15+3.50 [3.11, 3.89]+4.19+4.33
ODinW-13 (mean)50.4852.54+2.05 [1.49, 2.53]//
LVIS-VGD (point)41.9546.33+4.39 [3.86, 4.75]+5.34+5.57

Since both outputs come from the same query of the same model, the +0.97 to +4.39 AP gain (and up to +5.34 AP75 and +5.57 small object AP) comes from the box head itself rather than the backbone.

On the training side, Table 3 compares joint detection and segmentation training against segmentation-only training on the same Qwen3-VL-2B + DINOv3-S+ backbone. Adding box-only Objects365 data raises ADE20K open-vocabulary segmentation from 18.32 to 21.12 PQ (+2.80), 21.17 to 25.16 mIoU (+3.99), and 11.70 to 18.66 mask AP (+6.95). In our 12k-image Objects365 setting, ADE20K improves by +3.30 PQ, +4.91 mIoU, and +3.22 mask AP while COCO PQ stays at 56.41 vs. 56.42. We will revise Section 4.4 to state that Table 5 reports system-level results across all 14 tasks, whereas the fixed-backbone experiments above isolate the effect of direct box prediction and box-only joint training.

Q2. Multi-seed evaluation of the geometric consistency loss

The +1.5 box AP in Table 4 was measured on the DINOv3-S+ perceptor. We ran the same ablation on the final DINOv3-L perceptor with two new random seeds (1024 and 1025) under matched data and update budgets:

Seed (DINOv3-L Perceptor)Box AP (w/o → with loss)Δ Box APΔ Box AP75Δ Small object Box APMask outside box ↓
102455.23 → 55.56+0.34+0.08+0.504.26% → 2.61% (−38.7%)
102555.34 → 56.25+0.91+1.15+1.223.98% → 2.60% (−34.7%)

In both seeds, adding the loss improves box AP, AP75, and small object AP, and reduces the fraction of mask foreground pixels falling outside the predicted box by 35% to 39%. The box AP gain is smaller on DINOv3-L than on DINOv3-S+, and we will note this difference in the paper.

We then continued both pairs of checkpoints to the final multi-task stage (Qwen3-VL-4B + DINOv3-L) under the same data and update budget. At the final endpoint, overall COCO AP is unchanged (~56.1 to 56.3 box AP and ~51.6 to 52.0 mask AP), while small object mask AP increases by +0.43 and +0.18 and box AP75 by +0.05 and +0.06. Agreement between the predicted box and mask also improves in both seeds: mean box-to-mask IoU rises from 0.7514 to 0.7658 and from 0.7508 to 0.7691, and the fraction of mask pixels outside the box drops from 6.40% to 4.91% (−23.3%) and from 6.52% to 4.82% (−26.1%). We will include the training curves and clarify that the +1.5 AP gain in Table 4 is specific to the DINOv3-S+ perceptor setting.

Output interface

In Figure 2 and Section 3.2, every object query of the perceptor predicts a box and a mask in parallel, and the user's instruction selects which prediction is returned. We will make this clearer in Section 3.2 and the caption of Figure 2.

Thank you for raising the backbone confound in the X-SAM comparison. In Q1 of our rebuttal, we compared direct boxes with boxes derived from predicted masks using the same X-Chat checkpoint and selected queries. Direct boxes gained 0.97–4.39 AP across the four benchmarks. We’d appreciate your thoughts on whether this comparison helps clarify the contribution of direct box prediction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.