acceptodds
Withdrawn from ICLR 2027

MR2C: Multimodal Referring and Reasoning for Camouflaged Object Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Camouflaged Object Segmentation (COS) has achieved significant progress in recent years, yet existing methods mainly focus on vision-only mask prediction and remain limited in understanding flexible user intent in interactive scenarios. To extend the boundaries of COS and facilitate future research in this field, we propose Multimodal Referring and Reasoning Camouflaged Object Segmentation (MRR-COS), a new task that aims to segment the queried camouflaged object according to multimodal user references. To support this task, we construct MR2C, a multimodal benchmark containing 9932 images, 64 categories, 9932 segmentation masks, 19864 textual references, 1920 reference images, and 640 reference sounds. MR2C stands out with three key features: (1) four types of multimodal references, including referring text, reasoning text, reference images, and reference sounds; (2) an emphasis on understanding user intent and target-related sound content rather than merely detecting camouflaged regions; and (3) the incorporation of complex reasoning and world knowledge into its textual references. Furthermore, we introduce MCOSA, a multimodal camouflaged object segmentation assistant, to address multimodal understanding, reasoning, and fine-grained segmentation in MR2C. Extensive experiments show that MCOSA outperforms existing methods on MR2C, demonstrating the necessity of the proposed task and the effectiveness of our benchmark and framework. Our code and dataset will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Camouflaged Object Segmentation, Multimodal Referring, Reasoning Segmentation, Multimodal Large Language Models, Vision-Language

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x4oFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MRR-COS, a task for segmenting camouflaged targets using referring text, reasoning text, reference images, or reference sounds, and constructs MR2C with 9932 images covering 64 categories. Its MCOSA baseline combines multimodal token representations, segmentation-token refocusing, and query-conditioned local matching at multiple scales before mask decoding. Evaluation includes benchmark comparisons, component and training-data ablations, related segmentation tasks, reference perturbations, and an explanation-quality study involving 20 examples and five annotators.

Critical strengths
  1. MR2C provides a common camouflage evaluation setting for four reference types, with 19,864 textual references, 1,920 reference images, and 640 sounds, supported by manual checking procedures. MR2C’s useful contribution is bringing these reference forms together for segmentation in static camouflaged scenes.
  2. SUM offers a concrete architecture by normalizing a segmentation query’s attention over spatial positions within each local patch. With the backbone, decoder, and training schedule held constant, Table 12 reports multi-object mIoU of 59.4 for SUM versus 56.8 for MS-Fusion and 57.0 for MS-Attention, providing preliminary evidence for this matching design.
Critical weaknesses
  1. MCOSA’s refocusing equations use the same attention-weighted residual form and coefficient λf=0.1\lambda_f=0.1 already introduced by AnyRef. A possible extension from generated output tokens to all preceding tokens is neither clearly distinguished nor separately evaluated. Consequently, the manuscript does not establish a new refocusing mechanism, leaving SUM as its main identifiable architectural addition.
  2. The correspondence between category-level references and target masks is underdefined for multi-object scenes. Section 3 retains masks for the selected target while removing other annotated regions, but image and sound references are collected by category, and the supplied appendices do not explain how the selected mask is determined when several instances match that category. Figure 6’s plural bat references make this distinction relevant, because selecting one instance and segmenting all matching instances require different ground truth and imply different assessments of user intent.
Questions for authors
  1. What aggregation formula and evaluation sample counts reproduce the All, Single, and Multi scores in Table 3 from the test partition reported in Table 2?
  2. Does Eq. 5 aggregate only generated output tokens or also input tokens, and what substantive difference from AnyRef’s Eq. 2 is intended?
  3. When a reference image or sound matches several instances of the same category, does the ground-truth mask include all of them, and how is a subset specified when only some are selected?
  4. What training-data mixtures, sampling proportions, and numbers of updates were used for the adapted baselines in Table 3, particularly AnyRef and OISA, relative to MCOSA?
Reviewer_kQ1COpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes the Multimodal Referring and Reasoning Camouflaged Object Segmentation (MRR-COS), a task that segments the queried camouflaged object according to multimodal user references and generates a target-aware description. The authors construct MR2C, a benchmark of 9,932 images across 64 categories with pixel-level masks and four reference forms: referring text, reasoning text, reference images, and reference sounds. They introduce MCOSA, which unifies heterogeneous references into token representations, uses a refocusing mechanism to enrich the <seg> token representation, and applies SeekOut Matching (SUM) for multi-scale query–visual alignment. MCOSA is trained with text-generation and mask-prediction losses. In the Overall setting on MR2C, it achieves 69.5 mIoU and 46.4 METEOR, outperforming the compared specialist segmentation models and generalist MLLM baselines; results on related AVS and referring/reasoning segmentation benchmarks are also reported.

Critical strengths
  1. MR2C supports four reference forms—referring text, reasoning text, reference images, and reference sounds—within a single benchmark comprising 9,932 images with pixel-level masks across 64 categories, 19,864 textual references, 1,920 reference images, and 640 sound clips. Compared with vision-only COS, image-only Ref-COD, and language-only CamoQuery, MR2C provides broader coverage and establishes a unified evaluation platform for future research.

  2. MCOSA maps heterogeneous reference modalities into shared token representations and connects language reasoning with pixel-level segmentation through the <seg> token, refocusing, and SUM, outperforming the compared specialist and generalist baselines on MR2C.

Critical weaknesses
  1. The novelty and practical necessity of the proposed MRR-COS task are not sufficiently justified. Existing studies have already investigated reference-image-guided camouflaged object segmentation (e.g., Ref-COD) and reasoning-based camouflaged object segmentation (e.g., CamoQuery). Therefore, MRR-COS appears to primarily integrate existing referring and reasoning settings while extending them to additional reference modalities, rather than introducing a fundamentally new segmentation problem. More importantly, the paper does not adequately explain why these additional reference modalities are necessary or what unique challenges they introduce. For instance, in practical camouflaged object segmentation scenarios, it is unclear under what circumstances a user would prefer providing a reference sound over a simple textual description or reference image. Since the reference sounds are collected at the category level, they may merely serve as alternative representations of object categories rather than providing genuinely complementary information for target localization. Similarly, the necessity of reasoning-based references is not sufficiently demonstrated. Many reasoning expressions in the benchmark can seemingly be resolved into explicit object categories using general world knowledge, after which the task reduces to conventional category-guided segmentation. It remains unclear whether the proposed benchmark genuinely requires complex visual reasoning or simply combines category recognition with segmentation.

  2. The contribution of the proposed MR2C benchmark appears rather limited. As described in Section 3, the dataset is primarily constructed by reusing and reorganizing images from existing camouflaged object datasets, such as COD10K, CAMO, and NC4K, without collecting new camouflaged images or introducing substantially different visual scenarios. The main additional contributions consist of textual annotations and relatively small category-level reference pools, including 1,920 reference images and 640 reference sounds. Moreover, the segmentation masks are largely inherited from the original datasets, with modifications to retain the referred targets in multi-object scenes.

  3. The methodological novelty of MCOSA also appears limited. Its overall architecture largely follows existing MLLM-based segmentation frameworks, such as LISA and AnyRef, which already integrate multimodal understanding with pixel-level segmentation through dedicated segmentation tokens and mask decoders. The proposed Refocusing Mechanism closely resembles attention-weighted feature aggregation, where the <seg> token representation is enhanced using a weighted combination of preceding hidden states. Similarly, SeekOut Matching (SUM) relies on multi-scale local feature matching and aggregation, which are well-established techniques in visual segmentation.

Questions for authors
  1. Thirty reference images and ten sound clips are collected per category, but Table 2 gives only totals (1,920 and 640) without a train/test split for either reference pool. If the same category-level references are used in both splits, the reference-image and reference-sound results may partly reflect references seen during training. This matters for the 66.5% of images with multiple camouflaged objects: in the reference-image and reference-sound settings, a category-level reference does not indicate which same-category instance is queried, so it is unclear how much those results reflect instance-level grounding rather than category recognition.

  2. MMCSBench covers five tasks, including Image Grounding Segmentation (IGS), which provides 3,045 instance- and part-level masks; the other four tasks are text-only. Table 1 marks the Mask column for MMCSBench as “x” while listing 3,045 masks, and Section 2 describes the benchmark as lacking pixel-level annotations, so it would help to state that masks are limited to IGS. Table 3 also reports mIoU for a row labeled “MMCSBench” without identifying whether this is CamoVLM or another model, or how it was trained and evaluated on MR2C.

  3. The requested clarification concerns whether the image and sound reference pools are split into disjoint train and test sets; if not, held-out-reference results, same-category multi-target counts, and details on how the queried instance is identified are also needed.

  4. The “MMCSBench” row’s model and training protocol are unspecified; clarification should cover the model identity, whether the same MR2C split and recipe were used as for the other baselines (or same-protocol results), and how the 3,045 masks relate to Table 1 and Section 2.

Reviewer_c44eOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper extends COS from image-only mask prediction to multimodal referring and reasoning. It defines the MRR-COS task and introduces the MR2C benchmark, in which a model receives an image together with referring text, reasoning text, a reference image or a reference sound, and generates a description of the queried camouflaged object along with its pixel-level mask. On top of this benchmark, it proposes MCOSA, a baseline that feeds every reference modality into an LLM that emits a seg token, enriches the seg token through a refocusing mechanism, and matches it against multi-scale visual features with SeekOut Matching before decoding the mask. On MR2C, MCOSA outperforms specialist camouflaged object models and generalist multimodal LLMs. Ablations show that each component and the additional training data contribute, and the model also performs well on related segmentation benchmarks.

Critical strengths
  • The paper defines a new task in which the model segments the camouflaged object that a user points to, using referring text, reasoning text, a reference image or a reference sound. The commonly used COD benchmarks such as COD10K, CAMO and NC4K take only an image, so the new task tests whether the model follows what the user asks for and not only what looks camouflaged.
  • The baselines include both specialist camouflaged object models and generalist multimodal LLMs. The ablations add one component and one training data source at a time, so each gain comes from a single change and the reported results are easy to check.
Critical weaknesses
  • The paper does not say whether the reference images and the reference sounds are split into training and test sets. It also does not report how much the referring texts and reasoning texts vary within a category. A model trained on MR2C therefore cannot be trusted to generalize to references it has not seen.
  • The paper reports only mIoU and METEOR and omits standard COD measures such as MAE, S measure, E measure and weighted F measure, which makes the metric set narrow. Camouflaged objects have weak boundaries, so an overlap score can stay high even when the boundary is wrong. The problem is more serious here, since MR2C is a benchmark and its metrics set how later work is compared on it.
Questions for authors
  • Are the reference images and the reference sounds split into training and test sets?
  • How different are the referring texts and reasoning texts within one category?
  • Could you also report MAE, S measure, E measure and weighted F measure on MR2C? They are standard in COD, so the comparison with earlier camouflaged object methods would be direct.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.