acceptodds
Withdrawn from ICLR 2027

Adaptive Multi-Word Anchoring Network for Fine-Grained Image-Text Matching

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image-text matching bridges vision and language, which is a fundamental task in multimodal intelligence. The key challenge lies in establishing accurate alignment between two heterogeneous modalities. Existing methods typically rely on either dense soft attention, which suffers from cross-modal noise, or rigid hard assignment, which causes semantic information loss by artificially restricting region-word correspondences to one-to-one mappings. To address the inherent semantic density imbalance between visual and textual modalities, we propose an Adaptive Multi-Word Anchoring Network (MWAN). Specifically, we introduce an Anchored Alignment Mechanism (AAM) as a theoretically motivated adaptive gating approach tailored for cross-modal matching. By utilizing context-aware dynamic thresholds, AAM effectively grounds each image region to multiple semantically relevant words, achieving robust many-to-many alignment. Moreover, to overcome the distinguishability bottleneck in top-ranked retrieval results, we design a Residual Semantic Re-ranking (RSR) module that explicitly isolates and exploits previously suppressed residual information to discriminate among semantically similar candidates. Extensive experiments on Flickr30K and MSCOCO demonstrate that MWAN achieves superior performance. Notably, RSR also serves as a universal plug-and-play refinement module to enhance existing methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image-text matching, Semantic density imbalance, Anchored alignment mechanism

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VLYqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MWAN for fine-grained image-text matching. Its Anchored Alignment Mechanism (AAM) uses region- and text-conditioned sigmoid gates to associate each visual region with multiple words. Training incorporates visual and textual consistency losses, while Residual Semantic Re-ranking (RSR) refines the initial top-k results using low-saliency feature subsets. Experiments on Flickr30K and MSCOCO report improvements across several backbone configurations.

Critical strengths

The paper is clearly written and well organized. Preserving multiple relevant correspondences and exploiting additional details to distinguish similar candidates are reasonable motivations. The experiments cover multiple backbones and cross-dataset evaluation, with comparisons grouped by feature extractors. The fixed/global-threshold and loss ablations provide useful evidence for several design choices. Applying RSR to other matching models also suggests potential practical value.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for AAM’s proposed mechanism. The semantic consistency losses adopt standard InfoNCE formulations, making the conditioned gating and residual selection the main potential contributions. However, AAM’s textual conditioning is more restricted than its description might suggest. Eq. (3) can be rewritten as bj(T)=σ(wv⊤vj+wt⊤tˉ+c)=σ(aj+q(T)),b_j(T) = \sigma(w_v^{\top}v_j + w_t^{\top}\bar{t} + c) = \sigma(a_j + q(T)), where aj=wv⊤vj+ca_j = w_v^{\top}v_j + c and q(T)=wt⊤tˉq(T) = w_t^{\top}\bar{t}. Thus, the entire textual context contributes a single scalar q(T)q(T), shared across all regions. For a fixed image and trained model, any two regions jj and ℓ\ell satisfy logit⁡(bj(T))−logit⁡(bℓ(T))=wv⊤(vj−vℓ),\operatorname{logit}(b_j(T)) - \operatorname{logit}(b_{\ell}(T)) = w_v^{\top}(v_j - v_{\ell}), which is independent of the text. Consequently, changing the text shifts all threshold logits by the same amount and preserves the ordering of thresholds across regions. The region–word similarities remain text-dependent, but the additional textual conditioning in the threshold function is limited to this shared adjustment. Although Table 12 includes fixed and globally learned thresholds, it does not isolate the contribution of q(T)q(T) against a visual-only threshold bj=σ(wv⊤vj+c)b_j = \sigma(w_v^{\top}v_j + c). Nor does the paper directly establish that the learned thresholds track regional semantic complexity. The experiments therefore support the usefulness of adaptive weighting, while leaving the specific role of global textual conditioning and the stronger semantic-capacity interpretation insufficiently justified.

  2. RSR’s residual-semantic interpretation is not sufficiently established. PageRank centrality and syntactic saliency are not direct measurements of the evidence used by the original matcher. Moreover, removing positions from contextualized representations does not establish that dominant semantic information has been isolated. The reported gains and examples therefore do not distinguish recovery of previously suppressed semantics from the benefit of an additional, complementary score. This distinction matters because residual-semantic recovery is central to the module’s claimed contribution. The training objective for the learnable textual saliency branch is also insufficiently specified in Eq. (13) and Appendix B.

  3. Concrete theoretical and evaluation inconsistencies require clarification. The mutual-information identity in Eq. (37) does not hold in general, weakening the subsequent justification of the consistency losses as mathematically necessary or optimal. Separately, Table 11 reports Flickr30K image-retrieval R@5 increasing from 87.1 at k=0 to 87.2 at k=5. Under the same initial predictions, reordering only the top-five candidates preserves their membership and cannot change R@5. This discrepancy needs to be reconciled with the algorithm described in Section 3.3.

Questions for authors

1.Since the text contributes only a scalar shared across regions, how does this conditioning affect region–word alignment and lead to the reported performance gains?

  1. How can the R@5 increase at k=5 in Table 11 be reconciled with the stated procedure of only reordering the initial top-five candidates?
Reviewer_7kDzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes an Adaptive Multi-Word Anchoring Network (MWAN) for fine-grained image–text retrieval. Its main component, the Anchored Alignment Mechanism (AAM), assigns independent sigmoid gates to region–word similarities using a threshold conditioned on local visual features and global textual context. The gated similarities are normalized within each region and aggregated across regions using Log-Sum-Exp pooling.

Critical strengths

The experiments include both retrieval directions, MSCOCO 1K and 5K evaluations, multiple visual/textual backbones, and cross-dataset transfer. Grouping comparisons by backbone configuration is useful for interpreting performance.

Critical weaknesses

Appendix C.3 does not hold in general. Consider (R=U) as a uniformly distributed binary variable, (V=R), and (T) as a constant. Then (I(R;U)=1) bit, whereas both terms on the right-hand side are zero. Consequently, the presented derivation does not establish the subsequent claims about the necessity or mathematical optimality of symmetric consistency losses. Even a valid lower bound would not, by itself, establish that this particular pair of losses is optimal for retrieval.

The description of sparsemax in Appendix E.2 is also inaccurate. Sparsemax does not impose a predetermined number of selected words; its support depends on the input scores. Adaptive entmax likewise provides input-dependent sparsity. Therefore, fixed top-(c) selection is not an adequate substitute for these baselines.

The residual visual subset has a variable number of regions across candidates. Reusing the unnormalized LSE pooling of Eq. (11) therefore introduces a cardinality-dependent term.

Under Swin+BERT, RSR improves Flickr30K Recall Sum from 552.1 to 552.4 and MSCOCO 1K Recall Sum from 549.3 to 549.7. These gains are small, and no seed variability, confidence intervals, or paired significance analysis is reported. On a 1,000-image query set, a 0.1-percentage-point recall change corresponds to one query.

Questions for authors

Can the authors correct Eq. (37) and explain which conclusions about the consistency losses remain justified? How can Eq. (44) support the claimed inverted-U trend when its derivative is positive?

Which objective trains the dependency/POS embeddings and saliency MLP? How are gradients propagated through hard residual selection?

How does AAM compare with sparsemax, entmax, and normalized sigmoid attention under identical encoders, regularizers, and training budgets?

Reviewer_gfXuOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fine-grained image-text matching and proposes the Adaptive Multi-Word Anchoring Network (MWAN). The proposed method introduces an Anchored Alignment Mechanism (AAM) to enable multi-word alignment through adaptive region-specific thresholds, together with visual and textual semantic consistency losses to preserve intra-modal information. It further employs Residual Semantic Re-ranking (RSR) to refine retrieval results using residual semantic information. Experiments on Flickr30K and MSCOCO, including cross-dataset evaluations and ablation studies, demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The motivation is reasonable. The adaptive multi-word alignment design is intuitive and well motivated.
  2. The experimental validation is relatively comprehensive, covering multiple visual and textual backbones, MSCOCO 1K and 5K settings, cross-dataset evaluations, and relevant ablation studies.
Critical weaknesses
  1. The training procedures of several key components are not sufficiently described. In particular, the optimization of the learnable dependency/POS embeddings and saliency MLP, as well as the training of RSR, lacks sufficient implementation details, making the proposed method difficult to reproduce.
  2. Some theoretical derivations appear problematic. The mutual information identity in Eq. (37) does not generally hold, while Eq. (39) does not sufficiently establish the claimed mathematical optimality. Moreover, Eq. (44) appears to be monotonic in ρ under the stated conditions, which contradicts the claimed inverted-U relationship. These theoretical claims require further justification or revision.
  3. The independent contribution of RSR is not sufficiently validated, as its performance gains in Table 1 are relatively small and lack statistical analysis. In addition, the identical results for c=1 in Table 9 and CHAN in Table 1 raise questions about whether the Top-c comparisons are conducted under consistent training settings.
  4. The experimental evaluation mainly focuses on retrieval performance, without directly analyzing the region-to-word alignments learned by AAM. Therefore, whether the proposed mechanism actually captures the intended fine-grained semantic correspondences remains insufficiently validated.
Questions for authors
  1. Further clarification of how the dependency/POS embeddings and saliency MLP in Eq. (21) are trained is suggested, including their optimization objectives and whether they are retrained for different backbones. Additional implementation details of AAM, the consistency losses, and RSR are also needed to improve reproducibility.
  2. The derivations of Eq. (37) and Eq. (44) in Appendix C need further clarification or correction, particularly regarding the mutual information identity and the claimed inverted-U relationship. The validity of the associated theoretical conclusions should also be reconsidered.
  3. Further clarification of whether the Top-c experiments in Table 9 use identical training settings except for the alignment operator is suggested. Repeated-seed results or confidence intervals are also needed to verify the stability of the relatively small improvements introduced by RSR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.