- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper aims to improve vision-language representation with both global and local alignment for fine-grained and visually grounded understanding.
The two main ideas are: 1) unsupervised local alignment may be inaccurate, thus explicit supervision is needed; 2) current supervised local alignment relies on bounding boxes, which could be noisy, and thus pixel-level masks are needed.
The authors then construct such a dataset called MG-Data that contains paired visual-textual data at the image, object, and region level. 6 million images and image-level captions are sampled from CC3M and CC12M. 50 million object and region captions are generated by Qwen-VL Max through prompt engineering, which are further used to generate masks through MLCD-Seg.
Based on this dataset, the authors propose MaskALign, which combines mask-guided patch pooling, intra and cross-granularity alignment, and long-caption partial order (LPO) to improve discrimination of long captions over short captions.
The authors evaluate MaskAlign on box/mask classification, zero-shot semantic segmentation, long/short image-text retrieval, and zero-shot image classification. Ablations on method components and local visual feature construction strategies are also presented.
Critical strengths
The MG-Data dataset could be a valuable resource to the community, but it requires more effort in analysis and validation.
Critical weaknesses
-
Experiments do not support the benefits of explicit mask-guided alignment. The authors mentioned that the unsupervised/implicit alignment methods are "inaccurate", but comparisons with -CLIP show no consistent advantage despite additional dense supervision. MaskAlign performs worse on COCO bounding-box classification and Urban1K long-text-image retrieval. The authors mentioned that bounding boxes are noisy supervision, but comparisons with FG-CLIP are also unconvincing. MaskAlign trails FG-CLIP on LVIS bounding-box classification, while Table 5 shows only marginal gains from replacing box pooling with mask pooling.
-
MG-Data's annotation quality is insufficiently validated. MaskAlign uses explicit supervision from MG-Data and thus inherits the biases of the annotations. MG-Data relies on AI-generated captions and masks, yet the paper reports no human annotation-quality audit or quantitative assessment of attribute correctness, mask–caption correspondence, or region-description accuracy. This validation is particularly important if MG-Data is intended as a reusable resource for others to train or evaluate models, since annotation errors could affect both learned representations and the reliability of evaluation results.
-
The methodological novelty appears limited. Multi-granularity alignment is not new, and the proposed MaskAlign primarily combines mask-based pooling, standard contrastive objectives, averaged local embeddings, and an additional ranking loss. The paper needs to articulate and substantiate the new insight beyond this combination of existing techniques.
Questions for authors
-
How sensitive is the image-level short caption alignment? There are many ways to summarize object/region captions into short captions for the whole image, depending on the downstream task. Having a pre-determined short caption alignment target thus introduces additional biases, as well as training instability when the short caption is different from the average of the object/region captions.
-
Have the authors considered replacing the pre-determined short captions with a learned summarizing module from object/region level captions? The collection of object/region captions should already contain the information from all potential short captions. Making this process learned makes the method more scalable and better conforms to the data.
-
How is this paper different from distilling existing models’ capabilities? MG-Data sorely relies on pretrained VLMs and open vocabulary segmentors to generate detailed captions and spatial masks, suggesting that these models already possess the capabilities MaskAlign aims to learn. Transferring those capabilities into a more compact model can be valuable, but the paper should clarify what it contributes beyond distillation and demonstrate the resulting advantages in efficiency, generalization, or accuracy.
7 comments
W3: New insight beyond combining existing techniques.
Describing MaskAlign as a combination of existing techniques overlooks how it changes the training supervision. The key insight is that spatially grounded local alignment, incorporation of local semantics into global representations, and discrimination using additional long-caption details are related but distinct learning problems. MaskAlign introduces explicit supervision for each, rather than assuming that improving local alignment alone will automatically resolve the others.
1. Precise mask-based spatial support.
Box-based RoIAlign can include background and neighboring instances, while text-conditioned attention pooling infers spatial support from the text query. MaskAlign uses explicit mask coverage to select and average native patch features.
Mask-guided pooling improves zero-shot segmentation by 1.3–7.1 mIoU points over box-based RoIAlign and 2.5–16.1 points over β-CLIP-style attention pooling across all five benchmarks on both model families (Table 5). With the pooling operator held fixed, replacing bounding-box support with mask support improves zero-shot segmentation by 1.0–5.4 mIoU points. In Table 1, MaskAlign also improves bbox classification in most backbone-matched comparisons with the FG-CLIP family and β-CLIP. The strong results on region-based tasks indicate better token-level visual-textual correspondence.
2. Cross-granularity alignment.
Intra-granularity alignment supervises each scale separately, leaving the use of local semantics in global matching implicit. We therefore connect these granularities so that locally grounded details can improve the discriminative power of global representations. We pair aggregated object- and region-level visual features with image-level long captions, and aggregated local text features with global image embeddings.
Adding cross-granularity alignment on top of intra-granularity alignment improves DCI I2T/T2I R@1 by 3.6/5.0 points on CLIP and 1.4/0.8 points on SigLIP 2, while maintaining competitive performance on short-caption retrieval and zero-shot image classification (Table 4).
3. Long-caption Partial-Order (LPO).
Matching a long caption can rely on its overall scene semantics without fully exploiting the additional fine-grained details. LPO is designed to discourage this shortcut: it uses short-caption scores to identify hard negatives and requires a larger long-over-short similarity gain for matched pairs than for those negatives. In contrast, the triplet baseline separates matched pairs from hard negatives using long-caption similarities alone, without explicitly requiring the long caption to provide additional discrimination beyond the short-caption reference (Appendix B.1).
LPO improves DCI I2T/T2I R@1 by 0.6/0.8 points over triplet loss on both model families (Table 6). Adding LPO on top of intra- and cross-granularity alignment (#5 → #6) further improves DCI I2T/T2I R@1 by 1.3/1.1 points on CLIP and 0.7/1.1 points on SigLIP 2 (Table 4).
MaskAlign shows advantages on both region-based and holistic image understanding tasks. In Table 1, CLIP-based MaskAlign improves COCO box/mask classification by 7.4/8.2 points over FG-CLIP with ViT-B and outperforms β-CLIP in three of four dataset–scale settings on both classification tasks. With ViT-L, MaskAlign improves box classification over β-CLIP by 25.3 points on COCO and 27.6 points on LVIS. MaskAlign (SigLIP 2) outperforms FG-CLIP 2 on both COCO and LVIS at both scales. In Table 2, MaskAlign outperforms β-CLIP and FG-CLIP on all five zero-shot segmentation benchmarks. MaskAlign (SigLIP 2) also outperforms FG-CLIP 2 on all five benchmarks, with gains of 3.8–12.6 mIoU points.
In Table 3, both MaskAlign variants outperform their corresponding FG counterparts on all three long-caption retrieval benchmarks at both scales. For example, on DCI with CLIP ViT-L, MaskAlign achieves 65.3/67.3 I2T/T2I R@1, compared with 58.3/59.7 for FG-CLIP and 58.5/65.0 for β-CLIP. On Urban1K with ViT-L, MaskAlign (SigLIP 2) achieves 93.3/92.4, compared with 88.2/85.5 for FG-CLIP 2. MaskAlign remains competitive with its corresponding FG counterparts on short-caption retrieval and zero-shot image classification, and its CLIP variants outperform β-CLIP on every reported comparison for those tasks.
Overall, MaskAlign starts with mask-guided intra-granularity alignment to establish visual–textual correspondences at each scale, brings local semantics into global representations through cross-granularity alignment, and uses LPO to further improve long-caption discrimination. Across ViT-B and ViT-L variants of both CLIP and SigLIP 2, the framework delivers strong performance on both region-based and holistic image understanding tasks.
Q1: Sensitivity to fixed short-caption targets.
We use the original image-level captions from CC3M and CC12M as our short captions. In cross-granularity alignment, pooled object and region visual features are aligned with long captions, while pooled local text features are aligned with the global image embedding. Short captions are aligned with the global image embedding in the image-level objective.
In a preliminary experiment, aligning pooled local visual features with short captions reduced short-caption retrieval performance. One possible explanation is that short captions often capture the overall scene, whereas pooled local features emphasize object- and region-level details.
Q2: Replacing fixed short captions with a learned summarizer.
We interpret the proposed learned summarizing module as a text generator jointly trained with CLIP/SigLIP 2 to produce short captions from object and region descriptions that better support image–text alignment. We have not explored this approach. MG-Data’s short captions average 15.6 words, while object and region captions average 7.7 and 26.2 words, respectively. To produce a concise image-level caption, a summarizer would need to select which local details best characterize the image as a whole and learn to express them in a way that supports image–text alignment. Jointly training such a text generator would introduce additional computation and training complexity at MG-Data’s scale. The original CC3M and CC12M captions already provide a direct image-level training signal. Moreover, the extracted and filtered object/region descriptions may not retain all scene-level information, so they are not necessarily sufficient to reconstruct every suitable image-level short caption. For datasets with lower-quality short captions, summarizing the long caption would be an alternative worth investigating, since it provides broader scene context.
Q3: Contributions beyond distillation and resulting advantages.
Our contribution lies in how we use these annotations to supervise local and global vision–language alignment. Qwen and MLCD-Seg serve as annotation tools for constructing MG-Data, which is also intended as a reusable resource for future vision–language representation learning beyond MaskAlign. Tables 1–3 demonstrate the effectiveness of our framework on region-based and holistic image understanding tasks across CLIP and SigLIP 2 at both ViT-B and ViT-L scales, with the main results discussed in our response to W1.
Thanks for the review.
W1: Insufficient evidence for the benefits of mask-guided alignment.
Effect of explicit mask guidance.
In Table 5, mask-guided pooling improves zero-shot segmentation over RoIAlign, RLE-BBox, and β-CLIP-style attention pooling by 1.3–7.1, 1.0–5.4, and 2.5–16.1 mIoU points, respectively, across all five benchmarks on both model families. The RLE-BBox comparison holds the patch-pooling operator fixed, isolating the effect of spatial support. Mask-guided pooling also improves bbox classification in six of eight comparisons against the two box-based variants. Gains are concentrated on local tasks, with smaller changes in image-level performance, motivating cross-granularity alignment to explicitly incorporate locally grounded semantics into global representations.
Table 8 further shows that the original masks outperform both enlarged and shifted masks on COCO bbox classification and all five segmentation benchmarks for both backbones, supporting the importance of accurate spatial support for region- and patch-level semantic alignment.
1. Comparison with β-CLIP. β-CLIP leads by 1.4 points on ViT-B COCO bbox classification and 0.9 points on ViT-L Urban1K I2T. MaskAlign wins the other three bbox comparisons by 3.9 points on ViT-B LVIS and 25.3/27.6 points on ViT-L COCO/LVIS. β-CLIP also trails FG-CLIP in these three settings, suggesting that text-conditioned attention pooling’s effectiveness varies across datasets and model scales. MaskAlign wins 10 of 12 long-caption retrieval comparisons and all reported short-caption retrieval and zero-shot image classification comparisons against β-CLIP (Tables 1 and 3).
2. Comparison with the FG-CLIP family. We compare CLIP-based MaskAlign with FG-CLIP and SigLIP 2-based MaskAlign with FG-CLIP 2. MaskAlign wins 12 of 16 backbone-matched bbox/mask classification comparisons, leading on COCO with CLIP and on both COCO and LVIS with SigLIP 2. The exceptions are LVIS with CLIP, where its bbox accuracy trails by 2.1/0.9 points for ViT-B/ViT-L (Table 1).
One possible factor is training–evaluation consistency: FG-CLIP uses RoIAlign during both training and bbox evaluation, whereas MaskAlign trains with mask-selected patches and evaluates boxes using RoIAlign. This may favor FG-CLIP on bbox classification. Supporting this explanation, the CLIP ViT-B RoIAlign variant of MaskAlign in Table 5 achieves COCO/LVIS bbox accuracies of 50.4/25.0, compared with FG-CLIP’s 50.2/24.8 in Table 1. Thus, the observed ViT-B LVIS deficit disappears when both methods use RoIAlign during training and bbox evaluation.
Beyond bbox/mask classification, MaskAlign outperforms its matched FG counterparts on all five segmentation benchmarks (Table 2) and, at both model scales, on all three long-caption retrieval benchmarks, while remaining competitive on short-caption retrieval and zero-shot image classification (Table 3).
W2: Insufficient validation of MG-Data’s annotation quality.
1. Sampling. We conducted two rounds of human evaluation, one before full-scale dataset production and one after its completion. We randomly sampled 5,000 images per round, with no overlap, yielding 10,000 unique images in total. We evaluated all object- and region-level mask–caption pairs associated with these images. Both rounds evaluated annotations produced by the same finalized pipeline: the first served as a final validation before large-scale production, and the second as a post-production verification.
2. Evaluation protocol. In each round, five graduate students independently evaluated every sampled image and its associated annotations after a brief training session on the evaluation criteria and procedure. Each sample was presented in three views: the original image, an overlay of all object masks, and separate panels for the region masks. The evaluation covered four criteria: (1) factual accuracy of long captions, including whether the image supported the described details; (2) accuracy of object identities and attributes and their correspondence with the masked instances; (3) accuracy of region-level relations and contextual descriptions and their correspondence with the associated masks; (4) spatial coverage of the masks.
3. Results. All percentages are averaged across evaluators and rounds. The overall pass rate was 89.3%, with a pass requiring all four criteria to be met. Per-annotation pass rates were 97.1% for long captions, 95.3% for object annotations, and 94.2% for region annotations. Substantial under- or over-coverage occurred in 3.4% of object masks and 3.7% of region masks.