acceptodds
Withdrawn from ICLR 2027

Mask–Language Cycle Consistency for Test-Time Adaptation in Reasoning Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reasoning segmentation aims to predict pixel-level masks from complex, open-ended language queries. Despite recent progress, vision–language models remain vulnerable to test-time distribution shifts, such as environmental changes and image corruptions, which can disrupt mask–language alignment and degrade segmentation quality. Existing test-time adaptation (TTA) methods for segmentation primarily rely on entropy minimization or self-training, treating segmentation as a one-way prediction problem while overlooking its bidirectional cross-modal structure. We propose Cycle-ML, a TTA framework based on mask–language cycle consistency. At inference time, Cycle-ML predicts a mask from the input query, generates a referring expression grounded in the predicted mask, and regrounds the generated expression to obtain a cycle-reconstructed mask. It then uses both mask-level and language-level consistency as self-supervised adaptation signals. To mitigate error amplification from incorrect yet self-consistent predictions, we further introduce consistency–reliability learning, which estimates sample reliability from joint mask and language consistency and adaptively weights the resulting supervision. Experiments on multiple reasoning and referring segmentation benchmarks under 15 synthetic corruptions show that Cycle-ML consistently outperforms strong TTA baselines, demonstrating the effectiveness of reliability-aware mask–language cycle consistency for robust vision–language grounding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reasoning segmentation, Test-Time Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1dmbOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Cycle-ML, a test-time adaptation (TTA) framework built upon the UniPixel architecture. The method constructs an optimization loss based on the consistency of a mask-language cycle and uses it to adapt the model at test time for reasoning segmentation. To mitigate the issue of self-consistent but incorrect predictions, the authors further introduce a Consistency-Reliability Learning (CRL) mechanism, which jointly leverages cross-modal constraints to estimate prediction reliability and downweight unreliable samples during adaptation. Cycle-ML is evaluated on multiple benchmarks for reasoning segmentation and referring segmentation, covering 15 types of synthetic corruptions. Experimental results show that Cycle-ML generally improves performance over non-adaptive models and several representative TTA baselines.

Critical strengths
  1. Cycle-ML leverages UniPixel's ability to generate language descriptions from masks, which innovatively establishing mask-language consistency (MLC) and implementing consistency-reliability learning (CRL) within a bimodal paradigm.

  2. Extensive experiments, including various baseline methods and 15 types of synthetic corruptions, demonstrate that Cycle-ML serves as an effective test-time adaptation (TTA) strategy, significantly enhancing performance on reasoning segmentation and referring segmentation tasks.

Critical weaknesses
  1. Limited in Scalability and Generalizability

Does Cycle-ML require the base model to have the capability to generate text from mask? This paper does not justify why the reconstructed query must be generated by the same base model. If this is not essential, the authors should evaluate alternative query-generation strategies and include experiments on non-UniPixel architecture models to demonstrate the scalability and generalizability of Cycle-ML.


  1. Method's Defects

When the base model produces mutually consistent but incorrect masks and descriptions, the proposed method may assign an excessively high reliability weight to the adaptive loss. Specifically, it is possible to have Mori≈Mcyc≠Mgt,M^{ori} \approx M^{cyc} \neq M^{gt}, while simultaneously obtaining Cmask=IoU(Mori,Mcyc)≈1,Ctext=cos(zori,zcyc)≈1.C_{mask}=IoU(M^{ori}, M^{cyc}) \approx 1, C_{text}=cos(z_{ori}, z_{cyc}) \approx 1 . Consequently, the corresponding reliability weight would also be high, causing the model to reinforce incorrect predictions during adaptation. Figure 4 appears to show cases in which both mask and language consistency increase after adaptation, while the IoU with the ground-truth mask decreases substantially. Could this phenomenon be attributed to such self-consistent but incorrect predictions? The authors should provide a detailed analysis of these failure cases to better understand the limitations of the proposed reliability estimation mechanism.

Questions for authors
  1. Related Works

Could the authors discuss the following closely related TTA methods and clarify how Cycle-ML differs from them?

[1] (Seg-TTO) Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation

[2] (EATA) Efficient Test-Time Model Adaptation without Forgetting

[3] (ROID) Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correction

Could the authors clarify how Cycle-ML could be adapted to influential reasoning segmentation methods beyond UniPixel (such as those listed below)? What modifications would be required to apply Cycle-ML to these methods?

[4] (PixelLM) PixelLM: Pixel Reasoning with Large Multimodal Model

[5] (LENS) LENS: Learning to Segment Anything with Unified Reinforced Reasoning

[6] (Seg-Zero) Seg-Zero: Reasoning-Chain Guided Segmentation via Cognitive Reinforcement

[7] (SAM-R1) SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning


  1. Ablation Study

a. Could the authors specify the configuration of the CRL cold-start strategy mentioned in Appendix H.2, particularly the minimum buffer size required for “meaningful partitioning”? Could they also provide an ablation study examining how different cold-start configurations affect performance?

b. How sensitive is the final performance to the clipping strategy used to compute Gaussian reliability weights? Could the authors provide an ablation comparing the current strategy with removing clipping and using alternative clipping strategies?

c. How sensitive is Cycle-ML to the order in which test samples are processed? Could the authors evaluate multiple test-sample orderings and report the resulting performance variation to assess the robustness of test-time adaptation?


  1. Formatting Issues

Could the authors check and correct instances where en dashes are used within words that require hyphens?

Reviewer_Y58tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Cycle-ML addresses test-time adaptation for reasoning and referring segmentation under distribution shift. Starting from UniPixel, it predicts a mask, generates a mask-grounded referring expression, and segments the generated expression to reconstruct the mask. Mask agreement and language-feature agreement provide self-supervised adaptation signals. A consistency–reliability module groups buffered samples in a two-dimensional consistency space and applies Gaussian weights to the adaptation losses.

On ReasonSeg-test, average gIoU across 15 corruptions improves from 54.00 without adaptation to 56.84, compared with 55.17 for CoTTA. The paper also reports improvements on referring segmentation benchmarks, a 7B backbone, and cross-domain transfer to SegEarth-R1.

Critical strengths
  1. The adaptation objective is well motivated for multimodal grounding. The mask–language–mask cycle exploits information that conventional visual confidence objectives do not directly capture. The paper also explicitly addresses the important limitation that internally consistent predictions can still be incorrect.

  2. The empirical evaluation provides encouraging evidence across several settings. Results cover reasoning and referring segmentation, multiple corruption types, two backbone sizes, and an additional remote-sensing domain. The reliability ablation is particularly informative: unweighted cycle adaptation degrades performance, while reliability-aware adaptation recovers and improves it. The appendix provides detailed algorithms and discusses severe-shift failure cases.

Critical weaknesses
  1. The justification and necessity of the particular reliability estimator remain insufficiently established. Appendix A correctly distinguishes the normalized compactness objective from the unnormalized objective used in the main method, and acknowledges that the spectral rounding rule does not preserve discrete optimality. However, preserving the same Gram matrix does not establish comparable partitions. In particular, when pairwise affinities are nonnegative, the unnormalized objective permits a trivial all-in-one-cluster optimum. The implementation falls back to unit weights for degenerate partitions, potentially disabling the mechanism intended to prevent harmful adaptation. Since reliability weighting is essential to the reported gains, comparisons with direct consistency weighting, quantile selection, or two-dimensional k-means—and statistics on degenerate partitions—would help establish the value of the proposed estimator.

  2. The online evaluation protocol and computational trade-off need stronger characterization. Appendix H assumes a stationary stream with one corruption type per run, leaving robustness to mixed or changing shifts unresolved. Corruption severity, sample ordering, parameter/buffer resets, and repeated-run variability should also be specified more clearly. Appendix K explains that adaptation follows the current prediction, but this alone does not establish negligible latency impact on a resource-constrained stream: cycle generation and backward computation can delay subsequent samples. Actual throughput, peak memory, and wall-clock comparisons under a matched evaluation budget are needed to assess the claimed lightweight adaptation.

Questions for authors
  1. How does CRL compare with simpler reliability estimators using the same two consistency scores? How often does spectral partitioning degenerate, and what happens when most buffered cycles are self-consistent but incorrect?

  2. Please clarify the online scoring and update order, corruption severity, reset policy, and sensitivity to sample order. Can you report measured adaptation cost and performance under changing corruption streams?

Reviewer_rdDQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a cyclic-alignment mechanism wherein a query through a MLLM produces a mask, which in turn produces the query back, which can then be used to carry out language-alignment. The procedure remains symmetric for mask-consistency. The considered problem is Test Time Adaptation.

Critical strengths
  • Figure 1, in-domain vs out of domain test condition looks interesting.
  • equation 8 is also interesting too, the choice of keeping reliability as a vector instead of collapsing it into a scalar seems sensible
Critical weaknesses
  • The idea of cross modality alignment is not new. For example, please see cyclegan paper, https://arxiv.org/abs/1703.10593. Perhaps, the authors could elaborate a bit on if this idea is new in the domain of test-time domain adaptation itself.
Questions for authors
  • Fig 1, shows that mask space is corrupted by distribution shift. What happens if one corrupts language space? This can for example, occur on a voice channel when multiple speakers are speaking.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.