acceptodds
Under review as a conference paper at ICLR 2027

Language-Conditioned Mean-Field Unfolding: A Variational Inference Paradigm for Referring Expression Segmentation

Abstract

Deep algorithm unfolding bridges iterative optimization and differentiable neural architectures, yet its formulation has largely centered on recovering latent coefficients under predefined structural priors. In this work, we extend this paradigm to language-conditioned semantic inference, where the object to be unfolded is not an image-reconstruction coefficient, but a semantic posterior over a query-conditioned probability simplex. We propose Language-Conditioned Unfolding of Mean-Field Inference Networks (LUMIN), which formulates referring expression segmentation (RES) as variational mean-field inference in a text-conditioned energy model. LUMIN dynamically constructs a semantic state space from textual hypotheses together with an adaptive background state. A text-derived Gram compatibility kernel is coupled with a symmetric visual affinity graph to propagate language-conditioned evidence through simplex-constrained Softmax updates. Unrolling this inference process yields a compact and interpretable network in which each stage corresponds to a principled surrogate step on the variational free energy. A background-relative differential readout further calibrates marginal log-odds by suppressing encoder-induced common-mode drift. Experiments on RefCOCO/+/g demonstrate competitive segmentation performance and transparent step-wise posterior refinement with only three unfolding stages.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep Unfolding, Mean-Field Inference, Vision-Language Models, Text-Conditioned Energy Minimization, Semantic Posterior Inference

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_UN9aOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LUMIN, a language-conditioned mean-field unfolding framework for referring expression segmentation (RES). It formulates segmentation as variational inference over a query-dependent semantic state space, combining text-conditioned unary potentials, a language-derived compatibility kernel, and visual affinity to iteratively refine semantic predictions. The method further introduces a background-referenced differential readout and provides stage-wise optimization guarantees through majorization-minimization. Experiments on RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO demonstrate competitive zero-shot transfer performance, while ablation studies validate the contributions of the main components.

Critical strengths
  1. Novel and Theoretically Motivated Formulation. The paper introduces an interesting perspective by reformulating referring expression segmentation as language-conditioned variational inference rather than conventional feed-forward multimodal fusion. By integrating a text-derived compatibility kernel with visual affinity and mean-field unfolding, the proposed framework establishes an explicit connection between semantic inference and optimization. The stage-wise majorization-minimization guarantees further strengthen the theoretical foundation and interpretability of the method.

  2. Effective and Parameter-Efficient Posterior Refinement. The proposed framework achieves competitive segmentation performance under a zero-shot transfer setting while introducing relatively few stage-specific learnable parameters. The ablation studies demonstrate consistent improvements from iterative posterior refinement, particularly over the unary baseline, and support the effectiveness of the proposed semantic compatibility mechanism. These results suggest that structured inference provides a promising alternative to conventional multimodal refinement architectures.

Critical weaknesses
  1. Limited Validation of the Language-Conditioned Compatibility Mechanism. The proposed text-derived Gram compatibility kernel is a central contribution of the paper. However, in standard single-query RES, the compatibility matrix reduces to a diagonal form, eliminating interactions between different foreground semantic states. This raises concerns about whether the proposed semantic compatibility mechanism provides its intended benefits in the primary evaluation setting. Furthermore, the paper does not clearly establish how the multi-query semantic state space used during training relates to single-query inference, and lacks targeted ablations to clarify the effectiveness of this mechanism. This limitation weakens the empirical support for one of the paper's main contributions.

  2. Limited Fair Comparisons and Computational Efficiency Analysis. Although the paper reports competitive performance under a zero-shot transfer setting, the comparisons with existing RES methods involve different training protocols and pretraining datasets, making it difficult to assess the proposed framework's relative effectiveness. Furthermore, while parameter efficiency is emphasized as a key advantage, the construction of a dense visual affinity matrix introduces potentially substantial computational and memory costs. The paper lacks comprehensive end-to-end inference latency and memory comparisons under consistent hardware settings. These limitations weaken the experimental evidence supporting the practical advantages and efficiency of the proposed approach.

Questions for authors
  1. Effectiveness of the Semantic Compatibility Mechanism. In standard single-query RES, the text-derived Gram compatibility matrix reduces to a diagonal form, limiting interactions between different foreground semantic states. Could the authors clarify how the semantic state space is constructed during single-query inference and provide targeted ablation results demonstrating the effectiveness of the proposed compatibility kernel in this setting?

  2. Fairness of Experimental Comparisons. The proposed method is evaluated under a zero-shot transfer setting following large-scale multi-task pretraining, whereas several compared methods use different training protocols. Could the authors provide comparisons under more consistent training and evaluation settings, or additional experiments that better isolate the contribution of the proposed framework?

  3. Computational Efficiency and Practical Applicability. Although the method emphasizes parameter efficiency, constructing a dense visual affinity matrix may introduce substantial computational and memory overhead. Could the authors report the end-to-end inference latency, peak GPU memory consumption, and computational complexity, preferably with comparisons to representative RES baselines under identical hardware settings?

Reviewer_dEsYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Language-Conditioned Unfolding of Mean-Field Inference Networks (LUMIN), a variational inference framework for referring expression segmentation (RES). The proposed method formulates segmentation as continuous energy minimization over a language-conditioned probability simplex. Specifically, it constructs a text-conditioned unary field, a language-induced semantic compatibility kernel, and a symmetric visual affinity graph to model interactions between visual regions and semantic states. By introducing a positive semidefinite (PSD) completion strategy, the authors derive a Majorization-Minimization (MM) procedure and unfold the resulting mean-field updates into a three-stage inference network. A background-referenced differential readout is further introduced to produce the final segmentation predictions. Experiments on RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO demonstrate reasonable performance under the claimed zero-shot transfer setting.

Critical strengths
  1. The paper presents an interesting optimization-based perspective on referring expression segmentation. Instead of relying entirely on conventional feed-forward multimodal fusion, the proposed method formulates segmentation as language-conditioned variational inference. This provides an interpretable connection between structured prediction, mean-field inference, and deep unfolding.

  2. The mathematical formulation is relatively well organized. The authors construct a PSD visual affinity matrix and a text-derived Gram compatibility kernel, allowing the pairwise interaction to be incorporated into an MM optimization framework. The derivation of the entropy-regularized Softmax update provides a principled interpretation of the iterative inference process.

Critical weaknesses
  1. The methodological novelty is limited relative to existing CRF-based inference and deep unfolding methods. The main components, including visual affinity graphs, Gram-matrix compatibility kernels, entropy-regularized mean-field updates, and MM optimization, are established techniques. In particular, the iterative update resembles classical mean-field inference with language-conditioned compatibility weights. Although their combination is reasonable, the paper does not sufficiently clarify the fundamental methodological distinction from CRF-RNN and related structured inference frameworks. The authors should provide a more comprehensive comparison with existing mean-field unfolding and language-conditioned graph reasoning methods.

  2. The experimental performance is not sufficiently competitive, particularly on generalized referring expression segmentation. In Table 1, LUMIN does not outperform the strongest competing methods on any of the eight reported splits. More importantly, Table 2 shows substantial performance gaps compared with ReLA on gRefCOCO. For example, LUMIN achieves 52.3% cIoU on testA, compared with 69.3% for ReLA. The improvements in no-target accuracy on the validation split are encouraging, but they do not compensate for the substantial degradation in segmentation accuracy. These results weaken the empirical justification for the proposed framework.

The theoretical claim of vertex consistency requires further clarification. The proposed diagonal PSD completion introduces a self-interaction term into the continuous pairwise energy. However, the conventional discrete pairwise energy excludes diagonal interactions. Appendix A establishes consistency with a newly defined discrete self-interaction energy, rather than directly demonstrating equivalence to the original hollow-graph CRF energy. Furthermore, the diagonal entries of the compatibility matrix may differ between background and foreground states, making the added self-interaction term label-dependent. The authors should clarify in what sense the proposed relaxation preserves the original discrete energy and whether the additional diagonal terms alter the optimal discrete assignments.

  1. The relationship between the theoretical monotonic descent guarantee and the practical unfolding architecture is not sufficiently demonstrated. The proposed method uses stage-specific interaction weights and temperature parameters. Although the MM derivation guarantees energy descent within each stage-specific objective, it does not establish monotonic descent with respect to a single energy function across all unfolding stages. The authors acknowledge this distinction in Appendix C, but the main text still emphasizes monotonic optimization guarantees. Experimental energy trajectories and a comparison between tied and untied parameters would help substantiate the theoretical claims.

  2. The ablation studies are insufficient to isolate the contributions of the proposed components. The current experiments mainly examine inference depth, semantic compatibility, and readout mechanisms. However, the individual contributions of the adaptive background unary field, visual affinity graph, PSD completion, and learnable bandwidth are not adequately investigated. In addition, the heuristic propagation baseline produces 0.00% performance, which is an unusually degenerate result. Comparing against a collapsed baseline provides limited evidence of superiority over properly tuned conventional graph propagation or CRF-based refinement.

  3. The claimed zero-shot transfer capability and computational efficiency require stronger experimental evidence. The model is pretrained on approximately 3.37 million samples and uses pretrained visual and language representations. Therefore, the paper should provide a clearer analysis of possible overlap between pretraining data and evaluation benchmarks, as well as comparisons under matched pretraining resources and backbones. Furthermore, the visual affinity matrix is constructed over all pairs of spatial nodes, potentially introducing quadratic computational and memory complexity. Runtime, GPU memory consumption, and inference latency should be reported.

  4. The evaluation of multi-query inference and qualitative failure cases is limited. The framework explicitly constructs a dynamic semantic state space from multiple textual queries, yet most quantitative evaluations follow conventional single-query RES settings. The authors should provide experiments that systematically vary the number of simultaneous queries and evaluate semantic competition, overlapping targets, and multi-object segmentation. Additional failure cases would also help reveal the limitations of the proposed semantic propagation mechanism.

Questions for authors

see above mentioned

Reviewer_QBZzOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents language-conditioned unfolding of mean-field inference network for referring expression segmentation. In particular, the proposed LUMIN formulates RES as variational mean-field inference in a text-conditioned energy model. In addition, LUMIN dynamically constructs a semantic state space and an adaptive background state. The affinity graph is derived under textual condition. Each unfolded stage corresponds to a principled surrogate step on the variational free energy. A background-relative readout further improves performance. Experiments on RefCOCO/+/g proved the effectiveness of the proposed method.

Critical strengths

(1) The motivation is well presented of considering language-conditioned mean-field unfolding for referring expression segmentation.

(2) The illustrations are mostly clear and intuitive of the language-conditioned mean-field unfolding, the text-derived affinity graph, the simplex-constrained softmax update operator and the background-relative readout.

(3) The extension of classical mean-field unfolding into the text-conditioned setting sounds interesting for the community. Besides, the derivations are strict to the literature of graph learning and variational inference.

Critical weaknesses

(1) On Page 3, Line 153, the textual space is defined as an arbitrary set of M open-vocabulary textual queries. This may cause computational issues for the affinity graph when M is very large. There is no discussion on this point.

(2) On Page 5, Line 256, where \tao > 0 denotes the variational temperature controlling entropic dispersion. There is no explanation on how the value of \tao influences the framework. Also, there is no ablation on this hyperparameter.

(3) On Page 7, Sec 4.2 and Page 22 Table 6, the authors report the trainable params but ignores the full params and FLOPs. This is prone to unfair comparison if those important statistics are not compared against SOTA methods.

(4) On Page 8, Table 3, the ablation shows significant improvement of 1-stage mean-field over unary, while 2/3/4-stage shows little improvement. This arises a problem of the necessity of unfolding at the cost of 2/3/4-times inference cost.

(5) On Page 8, Line 414-415, the heuristic rule fails but listed in the ablation study. The reviewer and the experimental result is wield and the explanation is not convincing.

Questions for authors

No.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.