- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MWAN for fine-grained image-text matching. Its Anchored Alignment Mechanism (AAM) uses region- and text-conditioned sigmoid gates to associate each visual region with multiple words. Training incorporates visual and textual consistency losses, while Residual Semantic Re-ranking (RSR) refines the initial top-k results using low-saliency feature subsets. Experiments on Flickr30K and MSCOCO report improvements across several backbone configurations.
Critical strengths
The paper is clearly written and well organized. Preserving multiple relevant correspondences and exploiting additional details to distinguish similar candidates are reasonable motivations. The experiments cover multiple backbones and cross-dataset evaluation, with comparisons grouped by feature extractors. The fixed/global-threshold and loss ablations provide useful evidence for several design choices. Applying RSR to other matching models also suggests potential practical value.
Critical weaknesses
-
Limited methodological novelty and insufficient evidence for AAM’s proposed mechanism. The semantic consistency losses adopt standard InfoNCE formulations, making the conditioned gating and residual selection the main potential contributions. However, AAM’s textual conditioning is more restricted than its description might suggest. Eq. (3) can be rewritten as where and . Thus, the entire textual context contributes a single scalar , shared across all regions. For a fixed image and trained model, any two regions and satisfy which is independent of the text. Consequently, changing the text shifts all threshold logits by the same amount and preserves the ordering of thresholds across regions. The region–word similarities remain text-dependent, but the additional textual conditioning in the threshold function is limited to this shared adjustment. Although Table 12 includes fixed and globally learned thresholds, it does not isolate the contribution of against a visual-only threshold . Nor does the paper directly establish that the learned thresholds track regional semantic complexity. The experiments therefore support the usefulness of adaptive weighting, while leaving the specific role of global textual conditioning and the stronger semantic-capacity interpretation insufficiently justified.
-
RSR’s residual-semantic interpretation is not sufficiently established. PageRank centrality and syntactic saliency are not direct measurements of the evidence used by the original matcher. Moreover, removing positions from contextualized representations does not establish that dominant semantic information has been isolated. The reported gains and examples therefore do not distinguish recovery of previously suppressed semantics from the benefit of an additional, complementary score. This distinction matters because residual-semantic recovery is central to the module’s claimed contribution. The training objective for the learnable textual saliency branch is also insufficiently specified in Eq. (13) and Appendix B.
-
Concrete theoretical and evaluation inconsistencies require clarification. The mutual-information identity in Eq. (37) does not hold in general, weakening the subsequent justification of the consistency losses as mathematically necessary or optimal. Separately, Table 11 reports Flickr30K image-retrieval R@5 increasing from 87.1 at k=0 to 87.2 at k=5. Under the same initial predictions, reordering only the top-five candidates preserves their membership and cannot change R@5. This discrepancy needs to be reconciled with the algorithm described in Section 3.3.
Questions for authors
1.Since the text contributes only a scalar shared across regions, how does this conditioning affect region–word alignment and lead to the reported performance gains?
- How can the R@5 increase at k=5 in Table 11 be reconciled with the stated procedure of only reordering the initial top-five candidates?