acceptodds
Under review as a conference paper at ICLR 2027

Mask-Guided Multi-Granularity Contrastive Learning for Fine-Grained Vision–Language Alignment

Abstract

Contrastive vision-language models such as CLIP excel at global image-text alignment but remain limited in fine-grained understanding. Existing methods construct local visual evidence by either implicitly selecting text-relevant patches or extracting region features with box-based operations such as RoIAlign. The former lacks explicit spatial grounding, while the latter often includes background or neighboring objects. To address these limitations, we introduce MaskAlign, a multi-granularity vision-language learning framework centered on pixel-level mask supervision. Masks provide precise spatial support, enabling the model to aggregate native visual patches predominantly covered by the target while suppressing background-dominated patches. We further construct MG-Data, comprising 6M images with long captions of over 100 words on average, short captions, and approximately 36M object and 14M region annotations, each pairing a curated mask with a detailed description. Using MG-Data, MaskAlign establishes image-, object-, and region-level visual-textual correspondences and integrates locally grounded semantics into global representations through cross-granularity alignment. We also introduce Long-caption Partial-Order (LPO), which promotes the use of additional details in long captions by enforcing a larger long-over-short similarity gain for matched pairs than for mined hard negatives. Extensive experiments demonstrate strong performance across long- and short-text retrieval, zero-shot classification, and fine-grained localization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language Models, Fine-Grained Understanding, Multi-Granularity Learning, Mask-Guided Learning, Long-Text Understanding

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_auUTOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve vision-language representation with both global and local alignment for fine-grained and visually grounded understanding.

The two main ideas are: 1) unsupervised local alignment may be inaccurate, thus explicit supervision is needed; 2) current supervised local alignment relies on bounding boxes, which could be noisy, and thus pixel-level masks are needed.

The authors then construct such a dataset called MG-Data that contains paired visual-textual data at the image, object, and region level. 6 million images and image-level captions are sampled from CC3M and CC12M. 50 million object and region captions are generated by Qwen-VL Max through prompt engineering, which are further used to generate masks through MLCD-Seg.

Based on this dataset, the authors propose MaskALign, which combines mask-guided patch pooling, intra and cross-granularity alignment, and long-caption partial order (LPO) to improve discrimination of long captions over short captions.

The authors evaluate MaskAlign on box/mask classification, zero-shot semantic segmentation, long/short image-text retrieval, and zero-shot image classification. Ablations on method components and local visual feature construction strategies are also presented.

Critical strengths

The MG-Data dataset could be a valuable resource to the community, but it requires more effort in analysis and validation.

Critical weaknesses
  1. Experiments do not support the benefits of explicit mask-guided alignment. The authors mentioned that the unsupervised/implicit alignment methods are "inaccurate", but comparisons with β\beta -CLIP show no consistent advantage despite additional dense supervision. MaskAlign performs worse on COCO bounding-box classification and Urban1K long-text-image retrieval. The authors mentioned that bounding boxes are noisy supervision, but comparisons with FG-CLIP are also unconvincing. MaskAlign trails FG-CLIP on LVIS bounding-box classification, while Table 5 shows only marginal gains from replacing box pooling with mask pooling.

  2. MG-Data's annotation quality is insufficiently validated. MaskAlign uses explicit supervision from MG-Data and thus inherits the biases of the annotations. MG-Data relies on AI-generated captions and masks, yet the paper reports no human annotation-quality audit or quantitative assessment of attribute correctness, mask–caption correspondence, or region-description accuracy. This validation is particularly important if MG-Data is intended as a reusable resource for others to train or evaluate models, since annotation errors could affect both learned representations and the reliability of evaluation results.

  3. The methodological novelty appears limited. Multi-granularity alignment is not new, and the proposed MaskAlign primarily combines mask-based pooling, standard contrastive objectives, averaged local embeddings, and an additional ranking loss. The paper needs to articulate and substantiate the new insight beyond this combination of existing techniques.

Questions for authors
  1. How sensitive is the image-level short caption alignment? There are many ways to summarize object/region captions into short captions for the whole image, depending on the downstream task. Having a pre-determined short caption alignment target thus introduces additional biases, as well as training instability when the short caption is different from the average of the object/region captions.

  2. Have the authors considered replacing the pre-determined short captions with a learned summarizing module from object/region level captions? The collection of object/region captions should already contain the information from all potential short captions. Making this process learned makes the method more scalable and better conforms to the data.

  3. How is this paper different from distilling existing models’ capabilities? MG-Data sorely relies on pretrained VLMs and open vocabulary segmentors to generate detailed captions and spatial masks, suggesting that these models already possess the capabilities MaskAlign aims to learn. Transferring those capabilities into a more compact model can be valuable, but the paper should clarify what it contributes beyond distillation and demonstrate the resulting advantages in efficiency, generalization, or accuracy.

7 comments

Thanks for the review.

W1: Insufficient evidence for the benefits of mask-guided alignment.

Effect of explicit mask guidance.

In Table 5, mask-guided pooling improves zero-shot segmentation over RoIAlign, RLE-BBox, and β-CLIP-style attention pooling by 1.3–7.1, 1.0–5.4, and 2.5–16.1 mIoU points, respectively, across all five benchmarks on both model families. The RLE-BBox comparison holds the patch-pooling operator fixed, isolating the effect of spatial support. Mask-guided pooling also improves bbox classification in six of eight comparisons against the two box-based variants. Gains are concentrated on local tasks, with smaller changes in image-level performance, motivating cross-granularity alignment to explicitly incorporate locally grounded semantics into global representations.

Table 8 further shows that the original masks outperform both enlarged and shifted masks on COCO bbox classification and all five segmentation benchmarks for both backbones, supporting the importance of accurate spatial support for region- and patch-level semantic alignment.

1. Comparison with β-CLIP. β-CLIP leads by 1.4 points on ViT-B COCO bbox classification and 0.9 points on ViT-L Urban1K I2T. MaskAlign wins the other three bbox comparisons by 3.9 points on ViT-B LVIS and 25.3/27.6 points on ViT-L COCO/LVIS. β-CLIP also trails FG-CLIP in these three settings, suggesting that text-conditioned attention pooling’s effectiveness varies across datasets and model scales. MaskAlign wins 10 of 12 long-caption retrieval comparisons and all reported short-caption retrieval and zero-shot image classification comparisons against β-CLIP (Tables 1 and 3).

2. Comparison with the FG-CLIP family. We compare CLIP-based MaskAlign with FG-CLIP and SigLIP 2-based MaskAlign with FG-CLIP 2. MaskAlign wins 12 of 16 backbone-matched bbox/mask classification comparisons, leading on COCO with CLIP and on both COCO and LVIS with SigLIP 2. The exceptions are LVIS with CLIP, where its bbox accuracy trails by 2.1/0.9 points for ViT-B/ViT-L (Table 1).

One possible factor is training–evaluation consistency: FG-CLIP uses RoIAlign during both training and bbox evaluation, whereas MaskAlign trains with mask-selected patches and evaluates boxes using RoIAlign. This may favor FG-CLIP on bbox classification. Supporting this explanation, the CLIP ViT-B RoIAlign variant of MaskAlign in Table 5 achieves COCO/LVIS bbox accuracies of 50.4/25.0, compared with FG-CLIP’s 50.2/24.8 in Table 1. Thus, the observed ViT-B LVIS deficit disappears when both methods use RoIAlign during training and bbox evaluation.

Beyond bbox/mask classification, MaskAlign outperforms its matched FG counterparts on all five segmentation benchmarks (Table 2) and, at both model scales, on all three long-caption retrieval benchmarks, while remaining competitive on short-caption retrieval and zero-shot image classification (Table 3).

W2: Insufficient validation of MG-Data’s annotation quality.

1. Sampling. We conducted two rounds of human evaluation, one before full-scale dataset production and one after its completion. We randomly sampled 5,000 images per round, with no overlap, yielding 10,000 unique images in total. We evaluated all object- and region-level mask–caption pairs associated with these images. Both rounds evaluated annotations produced by the same finalized pipeline: the first served as a final validation before large-scale production, and the second as a post-production verification.

2. Evaluation protocol. In each round, five graduate students independently evaluated every sampled image and its associated annotations after a brief training session on the evaluation criteria and procedure. Each sample was presented in three views: the original image, an overlay of all object masks, and separate panels for the region masks. The evaluation covered four criteria: (1) factual accuracy of long captions, including whether the image supported the described details; (2) accuracy of object identities and attributes and their correspondence with the masked instances; (3) accuracy of region-level relations and contextual descriptions and their correspondence with the associated masks; (4) spatial coverage of the masks.

3. Results. All percentages are averaged across evaluators and rounds. The overall pass rate was 89.3%, with a pass requiring all four criteria to be met. Per-annotation pass rates were 97.1% for long captions, 95.3% for object annotations, and 94.2% for region annotations. Substantial under- or over-coverage occurred in 3.4% of object masks and 3.7% of region masks.

W3: New insight beyond combining existing techniques.

Describing MaskAlign as a combination of existing techniques overlooks how it changes the training supervision. The key insight is that spatially grounded local alignment, incorporation of local semantics into global representations, and discrimination using additional long-caption details are related but distinct learning problems. MaskAlign introduces explicit supervision for each, rather than assuming that improving local alignment alone will automatically resolve the others.

1. Precise mask-based spatial support.

Box-based RoIAlign can include background and neighboring instances, while text-conditioned attention pooling infers spatial support from the text query. MaskAlign uses explicit mask coverage to select and average native patch features.

Mask-guided pooling improves zero-shot segmentation by 1.3–7.1 mIoU points over box-based RoIAlign and 2.5–16.1 points over β-CLIP-style attention pooling across all five benchmarks on both model families (Table 5). With the pooling operator held fixed, replacing bounding-box support with mask support improves zero-shot segmentation by 1.0–5.4 mIoU points. In Table 1, MaskAlign also improves bbox classification in most backbone-matched comparisons with the FG-CLIP family and β-CLIP. The strong results on region-based tasks indicate better token-level visual-textual correspondence.

2. Cross-granularity alignment.

Intra-granularity alignment supervises each scale separately, leaving the use of local semantics in global matching implicit. We therefore connect these granularities so that locally grounded details can improve the discriminative power of global representations. We pair aggregated object- and region-level visual features with image-level long captions, and aggregated local text features with global image embeddings.

Adding cross-granularity alignment on top of intra-granularity alignment improves DCI I2T/T2I R@1 by 3.6/5.0 points on CLIP and 1.4/0.8 points on SigLIP 2, while maintaining competitive performance on short-caption retrieval and zero-shot image classification (Table 4).

3. Long-caption Partial-Order (LPO).

Matching a long caption can rely on its overall scene semantics without fully exploiting the additional fine-grained details. LPO is designed to discourage this shortcut: it uses short-caption scores to identify hard negatives and requires a larger long-over-short similarity gain for matched pairs than for those negatives. In contrast, the triplet baseline separates matched pairs from hard negatives using long-caption similarities alone, without explicitly requiring the long caption to provide additional discrimination beyond the short-caption reference (Appendix B.1).

LPO improves DCI I2T/T2I R@1 by 0.6/0.8 points over triplet loss on both model families (Table 6). Adding LPO on top of intra- and cross-granularity alignment (#5 → #6) further improves DCI I2T/T2I R@1 by 1.3/1.1 points on CLIP and 0.7/1.1 points on SigLIP 2 (Table 4).

MaskAlign shows advantages on both region-based and holistic image understanding tasks. In Table 1, CLIP-based MaskAlign improves COCO box/mask classification by 7.4/8.2 points over FG-CLIP with ViT-B and outperforms β-CLIP in three of four dataset–scale settings on both classification tasks. With ViT-L, MaskAlign improves box classification over β-CLIP by 25.3 points on COCO and 27.6 points on LVIS. MaskAlign (SigLIP 2) outperforms FG-CLIP 2 on both COCO and LVIS at both scales. In Table 2, MaskAlign outperforms β-CLIP and FG-CLIP on all five zero-shot segmentation benchmarks. MaskAlign (SigLIP 2) also outperforms FG-CLIP 2 on all five benchmarks, with gains of 3.8–12.6 mIoU points.

In Table 3, both MaskAlign variants outperform their corresponding FG counterparts on all three long-caption retrieval benchmarks at both scales. For example, on DCI with CLIP ViT-L, MaskAlign achieves 65.3/67.3 I2T/T2I R@1, compared with 58.3/59.7 for FG-CLIP and 58.5/65.0 for β-CLIP. On Urban1K with ViT-L, MaskAlign (SigLIP 2) achieves 93.3/92.4, compared with 88.2/85.5 for FG-CLIP 2. MaskAlign remains competitive with its corresponding FG counterparts on short-caption retrieval and zero-shot image classification, and its CLIP variants outperform β-CLIP on every reported comparison for those tasks.

Overall, MaskAlign starts with mask-guided intra-granularity alignment to establish visual–textual correspondences at each scale, brings local semantics into global representations through cross-granularity alignment, and uses LPO to further improve long-caption discrimination. Across ViT-B and ViT-L variants of both CLIP and SigLIP 2, the framework delivers strong performance on both region-based and holistic image understanding tasks.

Q1: Sensitivity to fixed short-caption targets.

We use the original image-level captions from CC3M and CC12M as our short captions. In cross-granularity alignment, pooled object and region visual features are aligned with long captions, while pooled local text features are aligned with the global image embedding. Short captions are aligned with the global image embedding in the image-level objective.

In a preliminary experiment, aligning pooled local visual features with short captions reduced short-caption retrieval performance. One possible explanation is that short captions often capture the overall scene, whereas pooled local features emphasize object- and region-level details.

Q2: Replacing fixed short captions with a learned summarizer.

We interpret the proposed learned summarizing module as a text generator jointly trained with CLIP/SigLIP 2 to produce short captions from object and region descriptions that better support image–text alignment. We have not explored this approach. MG-Data’s short captions average 15.6 words, while object and region captions average 7.7 and 26.2 words, respectively. To produce a concise image-level caption, a summarizer would need to select which local details best characterize the image as a whole and learn to express them in a way that supports image–text alignment. Jointly training such a text generator would introduce additional computation and training complexity at MG-Data’s scale. The original CC3M and CC12M captions already provide a direct image-level training signal. Moreover, the extracted and filtered object/region descriptions may not retain all scene-level information, so they are not necessarily sufficient to reconstruct every suitable image-level short caption. For datasets with lower-quality short captions, summarizing the long caption would be an alternative worth investigating, since it provides broader scene context.

Q3: Contributions beyond distillation and resulting advantages.

Our contribution lies in how we use these annotations to supervise local and global vision–language alignment. Qwen and MLCD-Seg serve as annotation tools for constructing MG-Data, which is also intended as a reusable resource for future vision–language representation learning beyond MaskAlign. Tables 1–3 demonstrate the effectiveness of our framework on region-based and holistic image understanding tasks across CLIP and SigLIP 2 at both ViT-B and ViT-L scales, with the main results discussed in our response to W1.

Reviewer_V14uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a multi-granularity vision-language learning framework, MaskAlign, for contrastive vision-language model training. Specifically, it introduces a new dataset with both long and short captions called MG-Data. During training, the framework builds image-, object-, and region-level visual-textual correspondences and cross-level alignment. The authors also introduce a long-caption partial-order (LPO) method for hard negative mining. They conduct experiments on different datasets and settings.

Critical strengths
  1. The paper is well written and easy to follow. The proposed method is technically feasible.
  2. The proposed dataset is helpful and may facilitate future research.
Critical weaknesses
  1. The novelty of utilizing multi-granularity vision-language learning is limited, as similar designs have been explored in previous works, such as UMG-CLIP, FG-CLIP, and β-CLIP.
  2. More analysis of the proposed dataset MG-Data is needed. This submission states that existing image–text datasets predominantly contain short captions. However, there is no comparison between the proposed dataset and previous datasets in terms of caption length and quality.
  3. The performance improvement of the proposed module is limited. In Table 4, the ablation studies show that the proposed cross-granularity alignment and LPO achieve limited gains. Adding cross-granularity to #4 only outperforms the baseline by 0.5% on the short retrieval case (I2T) and performs 0.9% worse on the ImageNet-1K case. Adding LPO leads to an improvement of 0.5% on the short retrieval case (I2T) and 0.1% on the ImageNet-1K case compared with #5.
Questions for authors

Please refer to the Critical Weaknesses section.

4 comments

Thanks for the review.

W1: Novelty over UMG-CLIP, FG-CLIP, and β-CLIP.

We would like to clarify the key contributions of MaskAlign beyond existing multi-granularity alignment methods.

1. Mask-guided local alignment.

Previous methods such as FG-CLIP and UMG-CLIP use box-based RoIAlign for local vision-language alignment. However, bounding boxes inevitably include irrelevant background or neighboring objects, introducing noise into local alignment. β-CLIP uses text-conditioned attention pooling to infer which patches are relevant to a description, so the pooled features can also include patches outside the target region. MaskAlign selects and averages native patch features based on mask coverage, reducing the contribution of background and neighboring instances. Mask-guided pooling improves semantic segmentation by 1.3–7.1 mIoU points over box-based RoIAlign and 2.5–16.1 points over β-CLIP-style attention pooling across all five benchmarks on both model families. It also improves Top-1 bbox classification over each of box-based RoIAlign and β-CLIP-style attention pooling in three of four backbone–dataset settings. Together, the bbox classification and segmentation results support stronger local semantic alignment at both the region and patch levels (Table 5).

2. Cross-granularity alignment.

Beyond aligning visual and textual features within each granularity, MaskAlign explicitly aligns aggregated object/region visual features with the global long-caption embedding, and aggregated object/region textual features with the global image embedding. These additional correspondences directly connect locally grounded semantics to both global modalities, rather than relying solely on their shared encoders to transfer local information. Adding this objective to the intra-granularity baseline improves DCI I2T/T2I R@1 by 3.6/5.0 points on CLIP and 1.4/0.8 points on SigLIP 2 (Table 4).

3.  Long-caption Partial-Order (LPO).

Matching a long caption can rely on its overall scene semantics without fully exploiting the additional fine-grained details. LPO is designed to discourage this shortcut: it uses short-caption scores to identify hard negatives and requires a larger long-over-short similarity gain for matched pairs than for those negatives. This encourages the additional details to contribute to discrimination, a requirement that conventional ranking on long-caption similarities alone does not explicitly impose. LPO improves DCI I2T/T2I R@1 by 0.6/0.8 points over triplet loss on both model families (Table 6). Adding LPO on top of intra- and cross-granularity alignment (#5 → #6) further improves long caption retrieval DCI I2T/T2I R@1 by 1.3/1.1 points on CLIP and 0.7/1.1 points on SigLIP 2 (Table 4).

MaskAlign shows advantages on both region-based and holistic image understanding tasks. In Table 1, CLIP-based MaskAlign improves COCO box/mask classification by 7.4/8.2 points over FG-CLIP with ViT-B and outperforms β-CLIP in three of four dataset–scale settings on both classification tasks. With ViT-L, MaskAlign improves box classification over β-CLIP by 25.3 points on COCO and 27.6 points on LVIS. MaskAlign (SigLIP 2) outperforms FG-CLIP 2 on both COCO and LVIS at both scales. In Table 2, MaskAlign outperforms β-CLIP and FG-CLIP on all five zero-shot segmentation benchmarks. MaskAlign (SigLIP 2) also outperforms FG-CLIP 2 on all five benchmarks, with gains of 3.8–12.6 mIoU points.

In Table 3, both MaskAlign variants outperform their corresponding FG baselines on all three long-caption retrieval benchmarks at both scales. For example, on DCI with CLIP ViT-L, MaskAlign achieves 65.3/67.3 I2T/T2I R@1, compared with 58.3/59.7 for FG-CLIP and 58.5/65.0 for β-CLIP. On Urban1K with ViT-L, MaskAlign (SigLIP 2) achieves 93.3/92.4, compared with 88.2/85.5 for FG-CLIP 2. MaskAlign remains competitive with its corresponding FG baselines on short-caption retrieval and zero-shot image classification, and its CLIP variants outperform β-CLIP on every reported comparison for those tasks.

Overall, across ViT-B and ViT-L variants of CLIP and SigLIP 2, MaskAlign consistently delivers strong performance in region-based and holistic image understanding tasks.

W2: Cross-dataset comparisons of caption length and quality.

ShareGPT4V, used by Long-CLIP and β-CLIP, provides image-level captions. FineHARD, used by FG-CLIP, additionally provides local captions paired with bounding boxes. MG-Data provides both object- and region-level captions paired with masks, enabling explicit spatial supervision for local alignment. The following table compares caption lengths, local annotation density, and truncation statistics across these datasets.

MG-Data provides richer local supervision, with 5.99 object-level and 2.27 region-level mask–caption annotations per image, compared with 3.37 bbox-level annotations in FineHARD. With maximum input lengths of 248 tokens for CLIP and 196 tokens for SigLIP 2 in our training, MG-Data discards fewer long-caption tokens than FineHARD: 2.7% / 7.9% versus 6.8% / 18.1%, respectively.

DatasetMean image-level caption length (words) (Long)Mean image-level caption length (words) (Short)Mean local annotations per image (Object)Mean local annotations per image (Region)Mean local caption length (words) (Object)Mean local caption length (words) (Region)Long captions truncated (CLIP / SigLIP 2)Long-caption tokens discarded (CLIP / SigLIP 2)
ShareGPT4V143.614.7--------5.9% / 31.1%1.3% / 6.1%
FineHARD198.516.93.37 (bbox)--2.91 (short); 7.08 (long)--38% / 74%6.8% / 18.1%
MG-Data155.515.65.99 (mask)2.27 (mask)7.726.212% / 37%2.7% / 7.9%

To assess annotation quality, we conducted a human audit involving five evaluators and 10,000 unique images across two rounds. The audit assessed caption factual accuracy, object identities and attributes, region-description correctness, mask–caption correspondence, and mask coverage. The reported pass rates were 97.1% for long captions, 95.3% for object annotations, and 94.2% for region annotations. Substantial under- or over-coverage occurred in 3.4% of object masks and 3.7% of region masks.

W3: Limited gains from cross-granularity alignment and LPO.

Cross-granularity alignment brings locally grounded object and region semantics into global representations for long-caption matching. Adding it (#4 → #5) improves long-caption retrieval DCI I2T/T2I R@1 by 3.6/5.0 points on CLIP and 1.4/0.8 points on SigLIP 2.

LPO operates on global image–caption scores. It uses short-caption scores as a reference and requires a larger long-over-short similarity gain for true matches than for hard negatives. Adding LPO (#5 → #6) further improves long-caption retrieval DCI by 1.3/1.1 points on CLIP and 0.7/1.1 points on SigLIP 2 (Table 4).

Adding cross-granularity alignment reduces CLIP’s ImageNet-1K accuracy by 0.9 points, while performance on the other tasks in Table 4 remains competitive. Table 3 puts this trade-off in context. MaskAlign outperforms its backbone-matched FG counterparts on all three long-caption retrieval benchmarks at both model scales, while remaining competitive on short-caption retrieval and zero-shot image classification. For example, with CLIP ViT-L, MaskAlign achieves 65.3/67.3 I2T/T2I R@1 on DCI, compared with 58.3/59.7 for FG-CLIP and 58.5/65.0 for β-CLIP. Its CLIP variants also outperform β-CLIP in every reported short-caption retrieval and zero-shot classification comparison, including 75.6% vs. 66.3% on ImageNet-1K with ViT-L.

We thank the reviewer for their constructive feedback and hope our rebuttal has addressed their concerns. If there are any new questions, we are glad to discuss and address them.

Dear Reviewer V14u,

Thank you again for your time and valuable feedback. As the discussion deadline approaches, we would be grateful if you could take another look at our responses and let us know whether they address your concerns.

If our responses have resolved your concerns, we would sincerely appreciate it if you could consider raising your score. We would be happy to provide any further clarification before the discussion closes.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_xpxVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MaskAlign provides a new pipeline for annotating mask-based images and training without sacrificing the precision of detection, and achieve SoTA in segmentation.

Critical strengths

Comprehensive study towards exploring a new technique for enhancing the segmentation ability of CLIP.

Critical weaknesses
  1. MaskAlign presents box-based supervision in prior work as a limitation and mask-based supervision as the solution, but box- and mask-based annotations may favor different tasks: box supervision may be more suitable for detection and localized recognition, while mask supervision may be more beneficial for dense segmentation. Thus, the paper should not imply that mask-based supervision is uniformly superior without clarifying the task-specific trade-offs (see Table 3's performance trade-off towards detection, where FG-CLIP perform better)

  2. The authors should revise the abstract and introduction to better motivate the work and state its contribution more precisely—for example, as a data annotation and learning pipeline for improving fine-grained representations beyond detection—rather than framing box-based supervision itself as the central limitation.

  3. In experiment. authors should make a more detailed compression with FG-CLIP, and make more analysis towards Table 5. To support MaskAlign's real contribution in "2."

  4. Please upload or paste the revised paragraph to me during rebuttal, the paper can be accepted if authors truly revise their article.

Questions for authors

Here, I ask Codex to find out the missing details to reproduce MaskAlign. I think this is the last chance that I can ask authors to provide their training and data recipe (If the work is accepted, I think it is really hard... due to business license)

Data

  1. How many images in the 6M total come from CC3M versus CC12M? Which specific dataset releases or snapshots were used?
  2. Please provide exact post-filtering statistics: number of images, short and long captions, object and region annotations.
  3. For long-caption generation, which exact Qwen-VL-Max model version or snapshot was used, and when were the API calls made? Please provide decoding settings such as temperature, top-p, maximum output length, and retry rules for failed calls. The manuscript gives the prompt but not these API settings.
  4. Please report quality-control procedures for generated annotations. Were captions checked for hallucinated details, object descriptions inconsistent with the image, incorrect region relations, or mask-caption mismatches? If so, provide the sample size, review criteria, and pass rate.
  5. How are model outputs converted into binary masks? Specify thresholds, resolution restoration, hole filling, and rules for selecting among multiple instances.
  6. Region masks are unions of object masks. How are invalid object references, and region annotations affected by object filtering handled?
  7. Exact data manifest and final annotation statistics.

Model

  1. What are the training and evaluation image resolutions, resize/crop procedures, interpolation methods, normalization means and standard deviations, and any random augmentations?
  2. From which vision-encoder layer and which patch tokens is the local feature map F taken? Is it passed through the final layer normalization, projection head, or L2 normalization? Are global and patch features represented in the same space?
  3. The paper says it uses Long-CLIP’s knowledge-preserving stretching strategy. Please specify the implementation/version, expanded text context length, and positional-embedding transformation. Is this also applied to SigLIP 2?
  4. Which model parameters are trainable or frozen? Are there additional projection layers, and how are they initialized? For the learnable temperature, specify its initial value, whether it is shared across granularities, and any bounds or constraints.

Training

  1. In addition to the learning rate, weight decay, and epoch count already reported, please provide scheduler, warm-up, and final learning rate; gradient clipping; parameter groups; and weight-decay exclusions.
  2. What are the actual optimizer-step count, drop_last policy, gradient accumulation procedure, random seeds, number of runs, GPU type/count, per-device batch size, and training time?
  3. For each granularity, over which set is the contrastive loss computed: the per-device batch, the global distributed batch, or the set aggregated by GradCache? Please specify the GradCache implementation, cross-device gathering, and temperature scaling.
  4. The manuscript uses a bidirectional contrastive objective and says the temperature is learnable. Do both CLIP and SigLIP 2 use the stated InfoNCE formula, or does SigLIP 2 use its native sigmoid loss? Please ensure the description, equations, and training code agree.
5 comments

Thanks for the review.

W1: Task-specific trade-offs between box and mask supervision.

We understand the reviewer’s comment to refer to bbox classification in Table 1. Table 3 reports image–text retrieval and zero-shot image classification.

For table 1, in the backbone-matched comparisons, CLIP-based MaskAlign is compared with FG-CLIP and SigLIP 2-based MaskAlign with FG-CLIP 2. MaskAlign family leads in six of eight box classification comparisons, with both deficits on LVIS with CLIP. The controlled comparison in Table 5 shows a similar pattern. Mask-based method improves box classification over bbox-based (RLE-pooling or ROIAlign) in six of eight settings and improves zero-shot segmentation on all five benchmarks for both model families. These results do not support a general characterization that mask-based supervision improves segmentation at the expense of bounding-box recognition.

One possible explanation for the LVIS results is that FG-CLIP uses RoIAlign during both training and bbox classification, whereas MaskAlign trains with mask-selected patches and uses RoIAlign for bbox classification. This consistency may favor FG-CLIP on bbox classification. Even with this discrepancy, on COCO with CLIP and on both COCO and LVIS with SigLIP 2, MaskAlign outperforms its corresponding FG baseline in bbox classification.

The RoIAlign variant in Table 5 supports this explanation. When both methods use box-based RoIAlign during training and bbox evaluation, MaskAlign (CLIP ViT-B) scores 50.4/25.0 on COCO/LVIS vs. FG-CLIP’s 50.2/24.8 (Table 1), eliminating the LVIS deficit.

During training, MaskAlign aligns mask-pooled features with object and region descriptions. Zero-shot segmentation tests patch-level alignment via ClearCLIP’s patch–text matching, while bbox classification tests pooled features from given boxes via RoIAlign, which FG-CLIP also uses during training. Together, the experimental results support improved local semantic alignment across both tasks.

W2: More comparisons with FG-CLIP and analysis of Table 5.

Comparison with FG-CLIP. We compare CLIP-based MaskAlign with FG-CLIP and SigLIP 2-based MaskAlign with FG-CLIP 2 at matched model scales. MaskAlign leads in 12 of 16 bbox and mask classification comparisons (Table 1). On COCO, it improves bbox/mask classification by 7.4/8.2 points over FG-CLIP with CLIP ViT-B and by 11.3/12.7 points over FG-CLIP 2 with SigLIP 2 ViT-B. It also improves zero-shot segmentation on all five benchmarks, by 1.7–6.8 mIoU points over FG-CLIP and 3.8–12.6 points over FG-CLIP 2 (Table 2). At the image level, MaskAlign leads the matched FG counterparts in all long-caption retrieval comparisons, while remaining competitive on short-caption retrieval and zero-shot classification (Table 3). We discuss the possible reason for the lower LVIS box classification scores with CLIP in our response to W1.

Further analysis of Table 5. With the training data and objectives held fixed, mask-guided pooling improves zero-shot segmentation on all five benchmarks for both model families, by 1.3–7.1 mIoU points over box-based RoIAlign and 2.5–16.1 points over text-conditioned attention pooling. The comparison with RLE-BBox holds the pooling operator fixed and still shows gains of 1.0–5.4 mIoU points. Mask guidance also improves Top-1 bbox classification accuracy in six of eight pairwise comparisons against the two box-based variants. Because zero-shot segmentation scores individual patches against category text, these results support stronger token-level visual–textual correspondence. As expected for a change to local supervision, the gains are concentrated in local tasks, with comparatively small changes on image-level tasks. Table 8 further shows that the original training masks outperform both enlarged and shifted masks on COCO bbox classification and all five segmentation benchmarks for both backbones, and remain competitive on LVIS. This supports the importance of accurate spatial support for both region-level and patch-level semantic alignment.

The COCO lead of text-conditioned attention pooling with CLIP ViT-B reverses sharply at a larger model scale and with a different backbone. In Table 1, β-CLIP leads MaskAlign by 1.4 points on COCO with CLIP ViT-B. With CLIP ViT-L, MaskAlign leads β-CLIP by 25.3 points on COCO and 27.6 points on LVIS. The controlled comparison in Table 5 shows the same sensitivity to backbone choice. β-CLIP-style pooling leads by 0.7 points on COCO with CLIP ViT-B, while mask-guided pooling leads by 11.3 points on COCO and 8.1 points on LVIS with SigLIP 2.

Mask-guided pooling directly averages mask-selected patches and outperforms the more complex Ferret-style extractor in nine of ten segmentation comparisons, with gains of up to 5.8 mIoU points (Table 5). Retrieval and box classification are broadly comparable overall, and Ferret-style extraction takes approximately 2–3× as long to train under the same settings.

Data

Q1: CC3M/CC12M composition and versions.

Of the 6,005,025 images, 808,429 (13.46%) are from CC3M and 5,196,596 (86.54%) are from CC12M. Each dataset has only one official release.

Q2: Post-filtering data statistics.

After filtering, the final MG-Data training set contains 6,005,025 images (each with one short caption and one long caption), 35,951,965 object annotations, and 13,652,729 region annotations. Each object or region annotation consists of a mask–caption pair.

Q3: Qwen version and generation settings.

We generated the long captions in late August 2025 using Alibaba Cloud Model Studio (Bailian), with the model ID qwen-vl-max-latest, which corresponded at that time to the qwen-vl-max-2025-08-13 snapshot following the August 14 update. We used the API defaults without explicitly overriding the decoding parameters: the default temperature was 0.01, while top_p and max_tokens were left unspecified. The model supported a maximum output length of 8,192 tokens. Failed requests were handled using the client SDK’s default retry behavior, with no additional retry logic.

In preliminary qualitative trials with GPT and other APIs, we observed no clear differences in caption quality for our annotation task. We therefore selected Qwen-VL-Max for its balance of caption quality and annotation cost.

Q4: Annotation quality control and validation.

1. Sampling. We conducted two rounds of human evaluation, one before full-scale dataset production and one after its completion. We randomly sampled 5,000 images per round, with no overlap, yielding 10,000 unique images in total. Both rounds evaluated annotations produced by the same finalized pipeline: the first served as a final validation before large-scale production, and the second as a post-production verification.

2. Evaluation protocol. In each round, five graduate students independently evaluated every sampled image and its associated annotations after a brief training session on the evaluation criteria and procedure. Each sample was presented in three views: the original image, an overlay of all object masks, and separate panels for the region masks. The evaluation covered four criteria: (1) factual accuracy of long captions, including whether the image supported the described details; (2) accuracy of object identities and attributes and their correspondence with the masked instances; (3) accuracy of region-level contextual descriptions and their correspondence with the associated masks; (4) spatial coverage of the masks.

3. Results. All percentages are averaged across evaluators and rounds. The overall pass rate was 89.3%, with a pass requiring all four criteria to be met. Per-annotation pass rates were 97.1% for long captions, 95.3% for object annotations, and 94.2% for region annotations. Substantial under- or over-coverage occurred in 3.4% of object masks and 3.7% of region masks.

Q5: Mask generation and post-processing.

We use the official MLCD-Seg API seg(image, caption, tokenizer, force_seg=True). The API restores the mask logits to the original image resolution using bilinear interpolation and padding removal, then assigns 1 to pixels with a logit > 0 and 0 otherwise. No hole filling is applied. With multimask_output=False, SAM returns the mask associated with its first mask token, yielding one binary mask per image–caption pair. We use this mask, without IoU-based ranking, non-maximum suppression, or any additional instance selection.

In our dataset post-processing (Supplementary Material, Section A.4), we discard masks covering more than 85% of the image. For mask pairs with IoU > 0.9, we randomly discard one mask. For pairs with IoU > 0.8, we discard the smaller mask if it has more than 10 connected components. We then apply morphological opening and remove object annotations whose masks select no patches for patch pooling. Each region mask is the union of the retained object masks associated with its description.

Q6: Handling invalid references and filtered objects in regions.

As described in Appendix A.4, object-level filtering is completed before region-level annotations are generated. We first filter and post-process the object masks, including removing object annotations whose masks yield no valid patches for patch pooling. We then provide the long image caption and only the retained object descriptions to Qwen-Max to generate region captions and their associated object sets. Each region mask is constructed as the union of the masks of its associated retained objects.

Q7: Data manifest and final statistics.

Please refer to our responses to Data Q1 and Q2 for the dataset composition and exact post-filtering annotation statistics. We promise to release MG-Data publicly upon acceptance.

Model

Q1: Image preprocessing and augmentation.

CLIP uses 224×224 images for both training and evaluation. Images are directly resized using bicubic interpolation; the subsequent 224×224 center crop leaves them unchanged. RGB values are scaled to [0,1] and normalized with mean (0.48145466, 0.45782750, 0.40821073) and standard deviation (0.26862954, 0.26130258, 0.27577711).

SigLIP 2 uses aspect-ratio-preserving NaFlex resizing with bilinear interpolation, 16×16 patches, and training batch-dependent patch budgets of {128,256,576,784,1024}. During training, images with either dimension below 128 are first resized to 512×512. Evaluation uses dynamic budgets for retrieval/local tasks and a fixed 1024-patch budget for ImageNet. RGB values are scaled to [0,1], with normalization mean and standard deviation both (0.5,0.5,0.5).

Neither model uses any random augmentations.

Q2: Local feature extraction and embedding space.

CLIP uses non-CLS tokens from the penultimate block, followed by the final block’s value/output projections and residual MLP, bypassing attention mixing, then final LayerNorm and the shared visual projection. SigLIP 2 uses final, post-LayerNorm patch tokens, excluding padding, followed by a dense head initialized from its pretrained pooling head with patch-wise queries. Both apply L2 normalization after mask pooling, rather than to individual patches. Global and local representations occupy the corresponding model’s shared image–text embedding space.

Q3: Context extension for CLIP and SigLIP 2.

Our local implementation follows Long-CLIP’s stretching strategy for both models: CLIP 77→248 tokens and SigLIP 2 64→196 tokens. At initialization, the first 20 positional embeddings are copied unchanged; subsequent positions undergo fourfold linear interpolation, with linear extrapolation at the tail. Short texts retain the original positional-embedding path. These embeddings remain trainable.

Q4: Trainable parameters, initialization, and temperature settings.

Both encoders and added heads are trainable; Long-text heads copy the pretrained text heads, and SigLIP 2’s dense head copies its pretrained multihead attention pooling head.

We parameterize the similarity scale as s = exp(logit_scale), with temperature τ = 1/s. The pretrained global initializations are approximately (s, τ) = (100, 0.01) for CLIP and (s, τ, b) = (115.302, 0.008673, −16.777) for SigLIP 2. Long/short objectives share global calibration. Object and region scales are separate trainable parameters, both initialized with log-scale 4.6052, corresponding to s ≈ 100 and τ ≈ 0.01, following FG-CLIP 2. CLIP’s cross-granularity terms reuse the corresponding object/region scales. No explicit temperature clamp is applied.

Training

Q1: Optimization settings.

Both models use AdamW with learning rate 3 × 10⁻⁶, weight decay 10⁻³, and cosine decay to zero. CLIP uses β = (0.9, 0.999), ε = 10⁻⁸, and 50 warm-up steps; SigLIP 2 uses β = (0.9, 0.98), ε = 10⁻⁶, and 3% training step warm-up. Gradient-norm clipping thresholds are 1000 and 10⁹, respectively, effectively disabling clipping for SigLIP 2. All parameter groups share the learning rate; LayerNorm parameters and biases are excluded from weight decay.

Q2: Training configuration and computational cost.

We trained each ViT-base model once with seed 42 on four NVIDIA A800 GPUs, using 1024 images per GPU (global batch 4096) for one epoch comprising 1466 optimizer updates. GradCache chunk sizes were 256/128 for CLIP/SigLIP 2, with drop_last=True and gradient_accumulation_steps=1. Training took approximately 8/18 hours.

Q3: Contrastive batch scope and GradCache implementation.

Our GradCache caches features, computes the complete logical-batch loss, and replays chunks with restored RNG states. Under this setting, all alignment branches use differentiable cross-device gathering over the complete logical batch, rather than individual chunks. Object/region objectives include all valid annotations, while cross-granularity objectives include valid aggregated pairs. Similarities are multiplied by the corresponding learned scale s, with an additional bias for sigmoid losses.

Q4: Clarify the loss functions used for CLIP and SigLIP 2 version of MaskAlign.

ObjectiveCLIPSigLIP 2
Global short/long alignmentBidirectional InfoNCESigmoid
Object/region alignmentBidirectional InfoNCEBidirectional InfoNCE
Cross-granularity alignmentBidirectional InfoNCESigmoid

The loss depends on both the backbone and the alignment objective, as summarized above. We will revise the manuscript’s descriptions and equations to explicitly distinguish these cases.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.