- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
X-Chat unifies detection and segmentation across seven task families through a dialogue-conditioned perceptor. Shared object queries predict boxes and masks, coupled by a geometric consistency loss, while box-only data support joint training. The paper also introduces LVIS-VGD and LVIS-INT detection evaluations and reports fourteen tasks with one checkpoint.
Critical strengths
-
This paper is clear and easy to follow.
-
Figures and tables are helpful for understanding the paper.
-
Supporting all 14 tasks with one ckpt is useful integration result, and the shared decoder directly addresses the fragmented output interface.
Critical weaknesses
-
The paper does not clearly show the benefits of training these tasks in one model over using an existing MLLM with perception tools. A direct comparison with such a system is missing. For example, the MLLM could interpret user instructions and call SAM 3 for detection and segmentation. The experiments do not establish better accuracy or inference efficiency over this alternative. This is a major concern because unification is the paper’s main contribution.
-
The comparison with X-SAM changes both the LLM and the perceptor. Shared box–mask queries also come from prior work (§2). Therefore, Table 5 does not isolate the benefit of joint detection–segmentation training. A comparison using the same backbone is needed to establish this benefit. This limits the evidence for the paper’s central contribution.
Questions for authors
-
On the same point-prompt split, how does direct box prediction compare with boxes extracted from predicted masks?
-
Does the interactive-segmentation drop persist in a matched-backbone comparison with and without detection training?
-
Could the authors report the inference latency for representative detection and segmentation requests using the shared model?
2 comments
Thank you for suggesting the Qwen3-VL + SAM 3 baseline. We implemented the pipeline you proposed and found that X-Chat gained 22.04 points in [email protected] on RefCOCO val. On 80-class COCO detection, median latency dropped from 2.79 s to 0.64 s at comparable AP. We’d be interested to hear how this comparison informs your assessment of the unified model’s practical value.
Thank you for noting the value of covering all 14 tasks with one checkpoint and a shared decoder.
Major concern & Q3. Comparison with an MLLM + SAM 3 pipeline and inference latency
We built a baseline pairing Qwen3-VL-4B-Instruct with SAM 3. For referring expressions, Qwen3-VL-4B-Instruct rewrites the user instruction into a target noun phrase for SAM 3, and we take SAM 3's top-scoring prediction. For generic detection, instance segmentation, and interactive prompts, inputs go directly to SAM 3 without an LLM call. Both systems were timed on the same single Ascend 910B2C device with batch size 1 and warm weights in memory:
1. Accuracy:
2. End-to-end latency and peak memory (batch size 1, 25 requests × 3 repeats per task):
Comparing the two systems shows where training a single model pays off:
Q1. Direct box prediction vs. boxes derived from masks on the same point split
On all 35,991 point requests of LVIS-INT val, using the same X-Chat checkpoint, forward pass, and selected query, direct box regression outperforms taking the bounding box of the predicted mask:
Repeating evaluation with three inference seeds gives +9.270 ± 0.006 box mIoU and +7.611 ± 0.055 [email protected]. On the 29,889 requests with nonempty masks, direct boxes still lead by +4.78 box mIoU because fragmented or noisy masks distort the enclosing box. The same comparison on fixed weights also favors direct boxes on LVIS-VGD point (+4.39 AP, +5.34 AP75, +5.57 small object AP), LVIS minival (+3.50 AP), ODinW-13 (+2.05 AP), and COCO (+0.97 AP).
Q2. Backbone alignment and interactive segmentation under joint training
We agree that Table 5 compares complete systems with different backbones rather than isolating joint detection and segmentation training, as noted in Section 4.4. Retraining an X-SAM baseline from scratch on Qwen3-VL-4B + DINOv3-L with an identical hidden-state normalization interface was not feasible within the rebuttal period. Instead, our fixed-backbone comparisons in Q1 (direct box vs. mask-derived box on the same checkpoint) and Table 3 (where adding box-only Objects365 data improves ADE20K segmentation by +2.80 PQ, +3.99 mIoU, and +6.95 mask AP on the same backbone) test both parts of our design under controlled conditions. We will revise Section 4.4 to separate the 14-task system comparison in Table 5 from these controlled ablations, and note the interactive segmentation gap to specialist models as a trade-off of training one model for all 14 tasks.