acceptodds
Withdrawn from ICLR 2027

LaMER: Language-Conditioned Multi-Expert Routing for Weakly Supervised Referring Expression Comprehension

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Referring Expression Comprehension (REC) localizes the image region described by a natural language expression, underpinning applications such as autonomous navigation, human-robot interaction, and medical image analysis. Under weak supervision, REC is challenged by the visual-linguistic semantic gap and the difficulty of inferring region-text correspondences from coarse image-level signals alone. Moreover, no single visual encoder captures all complementary properties—semantic alignment, fine-grained texture, and boundary awareness—while naively fusing multiple pretrained encoders yields heterogeneous, hard-to-reconcile representations. We propose LaMER, a weakly supervised REC framework that dynamically aggregates four frozen expert encoders through a language-conditioned Cross-Modal Routing Fusion module, adapting expert selection to each referring expression without instance-level bounding-box supervision. A Cell-Text Matching Head with Negative Anchor Augmentation, trained with a multi-objective loss combining contrastive, reconstruction, alignment, and load-balancing terms, further stabilizes optimization. LaMER achieves 77.67%, 62.81%, and 69.49% Acc@IoU\(\geq 0.5\) on RefCOCO, RefCOCO+, and RefCOCOg, surpassing DViN and WeakMCN by up to 16.49 and 14.49 points, with ablations validating each component.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Weakly Supervised Referring Expression Comprehension, Visual Grounding, Language-Conditioned Routing, Multi-Expert Learning, Cross-Modal Fusion, Vision-Language Alignment, Mixture of Experts, Referring Expression Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2qqjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a framework named LaMER for weakly supervised referring expression comprehension (WREC). LaMER introduces a Cross-Modal Routing Fusion module to integrate four types of visual encoders and adaptively combine their visual representations. Experimental results demonstrate that the proposed method achieves competitive performance compared with existing approaches.

Critical strengths
  1. The overall writing is clear and easy to follow. The paper is well organized, and the motivation and framework are presented in a relatively understandable manner.

  2. The proposed method achieves competitive performance on the evaluated benchmarks, demonstrating its effectiveness for WREC.

Critical weaknesses
  1. Although the paper claims the effectiveness of the Cross-Modal Routing Fusion module, I consider the overall contribution to be relatively limited. In Section 3, the description of existing methods occupies too much space, while the technical details and key innovations of the proposed method are insufficiently emphasized, making it difficult to clearly understand the main contributions of this work.

  2. The core idea of the Cross-Modal Routing Fusion module is to introduce a soft weighting strategy that learns four adaptive weights for combining different visual features. However, adaptive feature weighting and dynamic feature fusion strategies have already been widely explored in multimodal learning. Therefore, the authors need to further clarify the fundamental differences and novel aspects of the proposed module compared with existing feature fusion approaches.

  3. The ablation study is insufficient to validate the effectiveness of each component. It remains unclear which specific modules contribute to the performance improvement. More detailed ablation experiments and analysis are needed to demonstrate the necessity of each proposed component.

  4. The paper does not provide failure case analysis. Discussing failure cases would help reveal the limitations of the proposed method and provide deeper insights into its applicability and potential future improvements.

Questions for authors

see above mentioned

Reviewer_MwY2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LaMER, a language-conditioned multi-expert routing framework for weakly supervised referring expression comprehension (REC). The core motivation is that a single visual encoder cannot simultaneously model complementary visual cues such as semantic alignment, structure, texture, and boundary awareness. To address this, the method introduces four frozen visual experts in addition to the YOLO11x backbone, and designs a Cross-Modal Routing Fusion module that dynamically computes fusion weights for each expert. Experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate significant improvements over prior weakly supervised methods, and ablation studies validate the effectiveness of the multi-expert routing and fusion design.

Critical strengths
  1. Novel Multi-Expert Fusion with Dynamic Routing. The paper introduces a language-conditioned dynamic routing mechanism to integrate heterogeneous pretrained visual experts. This design effectively leverages complementary visual representations, including semantic, structural, and fine-grained information, enabling more adaptive and discriminative visual feature fusion for weakly supervised REC.

  2. Effective Region-Text Matching Strategy. The proposed Cell-Text Matching Head, enhanced by Negative Anchor Augmentation, improves the discrimination between candidate regions and referring expressions. Together with the multi-objective training strategy, it facilitates cross-modal alignment without requiring task-specific bounding-box supervision, contributing to the strong empirical performance.

Critical weaknesses
  1. Insufficient Ablation Studies and Fairness of Comparisons. The performance improvements may be partly attributed to the use of a stronger YOLO11x backbone, while some prior methods employ older backbones such as YOLOv5. Furthermore, the ablation studies do not sufficiently evaluate several architectural components and training objectives. Although the method is described as language-conditioned routing, the routing mechanism incorporates both visual and linguistic features, making it unclear how much improvement is specifically attributable to language guidance.

  2. Limited Methodological Novelty and High Computational Cost. The framework primarily combines established techniques, including multi-expert fusion, attention-based routing, and contrastive learning, which limits its methodological novelty. Moreover, the use of four additional pretrained visual encoders alongside YOLO11x introduces substantial computational complexity. Despite the strong performance, the paper does not provide sufficient analysis of training and inference costs. This makes it difficult to assess whether the performance gains justify the additional computational overhead.

Questions for authors
  1. Fairness of Comparisons and Ablation Studies. Since LaMER employs a stronger YOLO11x backbone while some previous methods use older architectures such as YOLOv5, how much of the reported performance improvement can be attributed to the proposed method rather than the backbone upgrade? Additionally, could the authors clarify the individual contributions of the proposed training objectives through further ablation studies?

  2. Effectiveness of Language-Conditioned Routing. Although the framework emphasizes language-conditioned expert routing, the routing query is constructed using both visual and linguistic features. Could the authors provide separate ablations to demonstrate the specific contribution of language guidance?

  3. Computational Complexity and Efficiency. The framework incorporates four additional pretrained visual encoders alongside YOLO11x, potentially introducing substantial computational overhead. Could the authors report the total and trainable parameters, inference latency, GPU memory consumption, and training cost? How do these computational requirements compare with existing methods, and are the performance improvements sufficient to justify the increased complexity?

Reviewer_x8XjOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents LaMER for weakly supervised referring expression comprehension. It combines four frozen visual encoders through language-conditioned routing and uses a Cell-Text Matching Head with Negative Anchor Augmentation for localization. The framework is trained with multiple objectives without bounding-box supervision for the grounding task. Experiments on RefCOCO, RefCOCO+, and RefCOCOg report substantial improvements over existing methods.

Critical strengths

Weakly supervised grounding is an important problem. Combining complementary pretrained visual features is a reasonable way to improve localization when bounding-box supervision is unavailable. The reported results are strong, particularly on RefCOCO and RefCOCO+.

Critical weaknesses

1-The core method offers limited novelty. DViN already addresses the limitations of a single visual backbone by selecting additional encoders and combining their features with learned weights. WeakMCN goes further by using softmax weights to dynamically fuse multiple encoders. LaMER adds language conditioning and QKV-based routing, but the paper does not clearly establish what this contributes beyond the existing fusion mechanisms. To me, this is an incremental change to an already explored approach.

2-The novelty of the Cell-Text Matching Head and NAA is unclear. RefCLIP already uses anchor–text similarity for localization and introduces Negative Anchor Augmentation. LaMER describes a closely related matching and negative-sampling scheme, yet lists this combination as a contribution without explaining what is new relative to RefCLIP. I do not see sufficient justification for treating this component as a separate methodological contribution.

Questions for authors

1-Top-N ablation. In Table 6, Top-1 achieves only 38.1% on RefCOCO validation, substantially below the YOLO11x baseline (67.2%) in Table 5, while Top-2 reaches 75.5%. Could the authors explain this large performance gap? How is the Top-1 expert selected, and does the YOLO11x backbone still contribute to the fused representation under this setting?

2-Computational efficiency. LaMER employs four pretrained visual encoders in addition to YOLO11x. Could the authors report the inference latency and GPU memory usage compared with existing methods such as DViN and WeakMCN, to better understand the computational cost of the reported improvements?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.