acceptodds
Under review as a conference paper at ICLR 2027

What Attention Aggregation Hides: Attention Entropy Routing for Referring Segmentation

Abstract

Language-guided segmentation requires a model to choose visual context that separates a referred region from nearby distractors. Existing methods mainly process Transformer features formed by aggregation weighted by attention, whose outputs do not explicitly record how concentrated the underlying attention distributions are. We introduce Attention Entropy Routing (AER), which treats normalized multiscale attention entropy as a complementary conditioning signal: language generates a bank of channel modulation vectors, local visual features and aligned entropy route among them, and pooled entropy conditions multiscale fusion. An analysis of single-head attention aggregation with fixed value vectors motivates this design, while heads trained on frozen visual features and controlled routing and spatial permutation studies test its effects at the task and spatial levels. Controlled comparisons show gains over visual routing from local entropy and further gains from global conditioning, while evaluations on RefCOCO, RefCOCO+, RefCOCOg, and remote sensing benchmarks with synthetic nonuniform haze cover natural and degraded visual settings.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Referring Image Segmentation, Vision-Language Learning, Attention Entropy, Conditional Routing, Vision Transformer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KqGjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Attention Entropy Routing (AER) for referring image segmentation. It extracts normalized attention entropy from four Swin Transformer stages, uses spatially aligned entropy to route language-generated channel modulations, and uses pooled entropy to condition scale fusion. A single-head proposition motivates entropy as information that an aggregated attention output need not retain. The evaluation covers three natural-image benchmarks and two remote-sensing benchmarks with synthetic haze. Within the proposed architecture, AER improves over visual routing by 2.44 mIoU points on hazy RefSegRS and 2.62 oIoU points on RefCOCO validation (Table 5a).

Critical strengths

1.The paper tests its central idea in several ways: a matched head with frozen encoder features (Table 4), local versus image-mean entropy routing (Table 5a), and a spatial permutation intervention (Table 5b). These controls provide more evidence than benchmark comparisons alone. The paper states the scope of its proposition carefully and reports cases where another method remains better on several RRSIS-D threshold metrics (Table 3).

Critical weaknesses

1.The experiments report one training seed (Appendix C.1) and use one fixed synthetic haze realization per source image (Appendix E.1). Several component gains are around one percentage point (Tables 4–5), with no run-to-run variation or evaluation on newly sampled haze. It is therefore difficult to judge their stability. 2.The gain attributed to global entropy fusion is not isolated from the added learned interaction parameters: AER adds U and VH when compared with the local-entropy configuration (Section 4.3; Appendix D). Also, the natural-image baseline scores come from separate publications rather than a shared retraining protocol (Section 5.1). Table 1 establishes a promising system-level comparison, but does not establish how much of its gap is caused by entropy.

Questions for authors

1.Can the authors report multiple-seed results, including variation for Tables 4–5, and evaluate the haze models on independently resampled haze? Were component and hyperparameter choices made without consulting RefSegRS test results? Can the authors add a parameter-matched global fusion control that replaces pooled entropy with a pooled visual descriptor? This would clarify whether the additional Table 5a gain comes from entropy rather than the larger scoring function.

Reviewer_pjPwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Attention Entropy Routing for referring image segmentation. The method uses normalized multiscale attention entropy as an additional conditioning signal to guide language-generated channel modulation and multiscale feature fusion. A theoretical analysis motivates the complementary role of attention entropy, while experiments on natural images and remote sensing datasets with synthetic haze demonstrate improvements over existing methods.

Critical strengths
  1. The paper introduces a well-motivated approach to incorporating attention concentration into language-guided segmentation. The combination of local entropy-conditioned routing and global entropy-guided fusion provides a coherent framework for adaptive feature processing.
  2. The experimental evaluation is comprehensive, covering natural images and degraded remote sensing imagery. The matched-head comparisons, component ablations, and spatial permutation experiments provide meaningful evidence for the complementary role of attention entropy.
Critical weaknesses
  1. Although the ablations demonstrate improvements from attention entropy, the paper does not compare it with alternative attention-distribution statistics under the same routing architecture. It remains unclear whether entropy provides distinctive advantages over other measures of attention concentration.
  2. The global fusion ablation introduces additional learned projections and a bilinear interaction alongside entropy conditioning. Consequently, the reported improvement cannot be attributed exclusively to global entropy without a capacity-matched control.
  3. The remote sensing experiments rely on a fixed synthetic haze generation protocol. The effectiveness of entropy routing under unseen degradation patterns, varying haze severity, or real atmospheric conditions remains insufficiently established.
Questions for authors
  1. Could the authors compare attention entropy with alternative attention concentration statistics, such as maximum attention weight or attention variance, under the same routing architecture?
  2. Could the authors provide a capacity-matched global fusion baseline using pooled visual features instead of entropy to isolate the contribution of global entropy conditioning?
  3. Could the authors evaluate AER under unseen haze patterns or severity levels to determine whether the benefits of entropy routing generalize beyond the current synthetic degradation protocol?
Reviewer_Ai7WOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Attention Entropy Routing (AER), using multiscale attention entropy to route language-generated channel modulation and condition scale fusion. A single-head nonidentifiability argument motivates the design. Evaluation covers three natural image datasets and two remote sensing datasets with synthetic haze, supported by extra component ablations.

Critical strengths
  1. Useful mechanism diagnostics. Frozen-encoder entropy conditioning improves scores by 0.59 points on hazy RefSegRS and 1.57 on RefCOCO. Local-versus-mean ablations and spatial permutation further support the usefulness of spatially aligned entropy.

  2. AER leads all 8 natural image splits among the listed methods. On hazy RefSegRS and RRSIS-D, test mIoU exceeds the strongest listed baselines.

Critical weaknesses
  1. The gains are not sufficiently attributable to entropy. Visual routing already reaches 77.21 RefCOCO val oIoU, versus AER's 79.83 and LAVT's 72.73 (Tables 1 and 5a). Moreover, local entropy receives four-stage information while its control uses current-stage features; global fusion adds an unmatched bilinear branch. These comparisons do not isolate entropy from other architectural gains, additional information, or capacity.

  2. Central experiments are insufficiently specified. Table 4 does not fully describe the language and conditioning paths or their trainable parameters. Appendix C.3 describes two output logits followed by sigmoid, whereas Eq. 28 uses one foreground probability. The exact probability mapping and implemented loss are needed to reproduce and interpret the reported gains.

  3. Clean remote-sensing evaluation is missing. Tables 2–3 evaluate only synthetic haze. Without results on the original RefSegRS and RRSIS-D, the paper does not establish whether the remote-sensing gains extend beyond the selected corruption distribution. Matched clean/hazy comparisons would clarify this scope.

Questions for authors
  1. How is the sentence representation pooled, and how do language and conditioning inputs enter Table 4's trainable head? Please specify the two-logit probability mapping and the exact implemented segmentation loss.

  2. Can Visual routing be reported across all natural-image splits? Do entropy gains persist with four-stage visual conditioning locally and a dimension-matched content conditioned bilinear branch globally?

  3. Are the gains in Tables 4 and 5(a) reproducible across seeds? Please report variability and specify the block size and permutation count used in Table 5(b).

  4. How do AER and its visual-routing control perform on clean RefSegRS/RRSIS-D? Can the authors clarify the distinction from AttEntropy and provide appropriately matched comparisons to omitted RIS methods?

  5. Why is synthetic haze a particularly informative test of the proposed hypothesis? Please provide matched clean/hazy comparisons to distinguish general architectural gains from benefits specific to degraded visual evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.