acceptodds
Under review as a conference paper at ICLR 2027

TAG: Calibrating SAM3 for Universal Image Segmentation without Dense Supervision

20 views

Abstract

The recent Segment Anything Model 3 (SAM3) extends powerful promptable segmentation from spatial cues to concepts, raising the prospect of a single foundation model for universal image segmentation. However, this promise often fails to translate into reliable performance on downstream segmentation tasks. In this work, we empirically identify concept recognition, rather than mask quality, as the primary bottleneck: SAM3 accurately delineates the objects it recognizes yet struggles to robustly determine which concepts are present. Driven by this, we present Text Anchor re-Grounding (TAG), a simple yet effective framework that rectifies the concept recognition of SAM3 on the target domain without dense annotations. Specifically, TAG introduces an Anchor Residual Pathway comprising image-conditioned, low-rank adapters to re-ground text anchors in the target distribution and modulate subsequent text-conditioned interactions. Guided solely by image-level category labels, TAG jointly optimizes binary cross-entropy with Soft Boundary Ranking objectives to calibrate presence confidence and distinguish present concepts from confusable negatives. Building upon this, we devise Confidence-Weighted Distillation to couple rich semantic context of a multimodal large language model with fine-grained spatial priors of SAM3, thereby achieving effective calibration in the absence of manual annotations. Extensive experiments demonstrate that TAG unlocks SAM3's segmentation potential with only 3.1M trainable parameters, achieving impressive performance across diverse scenarios, and notably, even surpassing leading mask-supervised methods on several metrics. Our code will be publicly available. Anonymous github link: https://anonymous.4open.science/r/TAG-9038.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Universal image segmentation, Segment Anything Model 3, Weakly supervised learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gjsjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TAG adapts a frozen SAM3 to downstream universal segmentation (semantic, instance, panoptic) using only image-level class labels. Oracle interventions are used to argue that SAM3's downstream failures are concept-recognition errors rather than mask errors. A 3.1M-parameter Anchor Residual Pathway (low-rank and image-conditioned residuals on text features, fusion-encoder text K/V, decoder image K/V, and the presence token) is trained with BCE on presence plus a pairwise hard-negative ranking loss. TAG-V replaces human labels with Qwen3.8-27B presence probabilities through confidence-weighted distillation. On ADE20K, Cityscapes, and iSAID, TAG-I improves frozen SAM3 by +17.2 PQ, +9.5 AP, and +20.8 mIoU on average (3 seeds, one shared configuration); TAG-V by +14.0, +7.5, and +16.8. Appendices add VOC and COCO results, comparisons with training-free SAM3 calibration methods, LoRA and presence-classifier controls, and hyperparameter sweeps.

Critical strengths
  1. Large, consistent gains at low cost. With the backbone frozen, one shared hyperparameter set, and 3-seed averaging, TAG improves frozen SAM3 by 12 to 25 PQ across natural, urban, and aerial domains. This is a practically useful recipe for deploying SAM3 on a closed vocabulary without masks.

  2. Controls isolate where the gain comes from. Output-level presence calibration (class-wise affine, image-based and token-based classifiers) and a budget-matched LoRA are all clearly below TAG-I, showing that adapting how text conditions the model matters more than re-scoring presence on fixed features.

Critical weaknesses
  1. The baselines that most directly threaten the contribution are missing. For TAG-V, gating frozen SAM3 at test time with the same Qwen3.8-27B presence predictions (or SAM 3 Agent) is the natural reference; if it matches TAG-V, the contribution reduces to efficiency. For TAG-I, prompt or class-embedding learning on a frozen VLM with the same BCE plus ranking objective (CoOp/CoCoOp style; Parisot et al., CVPR 2023, learn class-name embeddings on a frozen model for exactly this purpose) is the natural reference; the budget-matched LoRA adapts only cross-attention K/V and is not a substitute. Without these, it is unclear whether ARP's specific design matters or whether any text-side adaptation with this objective would do.

  2. Claims outrun the evidence. The "GT categories" oracle assigns each predicted component its majority GT label, which injects GT spatial information, and the "GT mask regions" oracle scores below frozen SAM3 (22.3 vs. 37.5 mIoU) without explanation. The natural ceiling for image-level supervision, keeping GT-present classes and suppressing absent ones, is not reported, so the "recognition, not masks" diagnosis is asserted more strongly than shown. On VOC and COCO, frozen SAM3 alone already beats every weakly supervised baseline (VOC 83.5 vs. 82.6 mIoU; COCO 43.9 vs. 25.2 AP), yet App. D.2.3 attributes the full margins to TAG. SAM3 is also pretrained on masks from adjacent domains (BDD100K and VisDrone are in SA-Co/EXT), so "without dense supervision" holds only for the target dataset, and "surpassing fully supervised methods" is true for two selected cells.

Questions for authors
  1. How does frozen SAM3 perform when gated at inference by the same MLLM presence predictions used to train TAG-V?

  2. How does a CoOp/CoCoOp-style or Parisot-style class-embedding baseline perform under your BCE plus SBR objective?

  3. How is a category assigned to each region in the "GT mask regions" oracle, and what does the full GT-presence oracle (present classes set to 1, absent classes set to 0) score?

  4. What are the latency and memory per image on ADE20K, given per-class fusion and decoder passes over 150 classes?

Reviewer_ZLB5OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the limitations of SAM3 on downstream universal image segmentation and identifies concept recognition, particularly inter-class confusion and under-activation, as a major bottleneck. Based on this observation, the authors propose Text Anchor re-Grounding (TAG), which keeps SAM3 frozen and introduces an Anchor Residual Pathway (ARP) to adapt text features and text-conditioned interactions using image-level category labels. The training objective combines binary cross-entropy with Soft Boundary Ranking (SBR). The authors further introduce Confidence-Weighted Distillation (CWD) to train TAG using image-level pseudo-labels generated by an MLLM. Experiments on ADE20K, Cityscapes, and iSAID show substantial improvements over frozen SAM3 across semantic, instance, and panoptic segmentation.

Critical strengths

Interesting problem formulation and motivation. The observation that concept recognition, rather than mask quality, is a major limitation of SAM3 is interesting. The oracle experiments and object-level analysis provide useful evidence for this hypothesis. Using image-level supervision to improve segmentation without dense annotations is a well-motivated direction.

Strong experimental results across different segmentation tasks. TAG achieves substantial improvements over frozen SAM3 on ADE20K, Cityscapes, and iSAID, covering semantic, instance, and panoptic segmentation. The results are particularly encouraging given that SAM3 remains frozen and only 3.1M additional parameters are trained.

Useful ablations supporting the adaptation strategy. The authors provide comparisons with ordinary LoRA, text-only adaptation, and presence-score calibration, along with ablations of the individual ARP components and training objectives. These results help establish that adapting the text-conditioned pathway is more effective than simply correcting the final presence scores

Critical weaknesses

Insufficiently Motivated Method Design. While the identified problem is interesting and well-motivated, the proposed solution appears somewhat ad hoc. ARP combines several adaptation mechanisms, including text-feature residuals, image-conditioned cross-attention, fusion/decoder K/V updates, and presence-token initialization. Although the ablation studies demonstrate the effectiveness of these components, the paper does not sufficiently explain why this particular combination is needed to address the identified recognition failures. The overall design feels more like a collection of incremental modifications than a well-justified framework built around a clear technical insight.

The contribution of concept recognition correction is not fully isolated. The oracle experiments provide evidence that correcting category assignments and activations can significantly improve segmentation. However, TAG also modifies the text-conditioned interactions in the fusion encoder and decoder, which can change the predicted masks themselves. It remains unclear how much of the improvement comes from better concept recognition versus changes in the mask predictions. A more direct analysis separating these effects would strengthen the central claim.

The benefit of MLLM-based distillation is not sufficiently isolated. TAG-V uses an MLLM to generate image-level category presence labels, which are then used to train ARP. While the paper compares CWD with naive BCE, it does not clearly establish how much of the improvement comes from learning ARP rather than transferring the stronger category recognition capability of the MLLM. A comparison using the same MLLM-generated labels to train a simpler presence-score calibrator would make the contribution of TAG-V clearer.

Questions for authors

Can the authors quantify how much of the improvement comes from correcting presence scores and category assignments versus changing the predicted masks? For example, evaluating different combinations of frozen SAM3 and TAG masks and presence scores would help clarify the underlying mechanism.

For TAG-V, how would a simpler presence-score calibration method trained with the same MLLM-generated image-level labels compare with the full ARP? It would also be interesting to compare with directly using MLLM presence predictions to filter frozen SAM3 outputs, while accounting for the additional inference cost.

The paper argues that image-level supervision can effectively address inter-class confusion. However, when two visually similar categories are both present in an image, BCE and SBR treat both as positive classes without providing any spatial information to distinguish them. How does TAG resolve such cases, and to what extent can image-level supervision address this type of confusion?

Reviewer_WCbLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes framework for adapting SAM3 to universal image segmentation without target-domain dense annotations. The authors first identify the bottleneck of SAM3 through empirical analysis: concept recognition is more important than mask quality. Based on this observation, the method introduces an Anchor Residual Pathway (ARP) to calibrate text features. The model is trained using category labels with a proposed Soft Boundary Ranking (SBR) objectives. Confidence-Weighted Distillation (CWD) further enables training on unlabeled images using MLLM-generated pseudo-labels. Experiments across multiple benchmarks show substantial improvements. Extensive ablation studies are provided.

Critical strengths

The paper addresses the practical problem of adapting large segmentation foundation models without dense annotations. The authors provide an interesting empirical diagnosis of SAM3's failures and propose a targeted adaptation framework. Although the individual techniques are not entirely new, the SAM3-specific design of ARP still represents a meaningful contribution. The experiments are comprehensive, covering semantic, instance and panoptic segmentation. Both human-supervised TAG-I and MLLM-supervised TAG-V achieve noticeable improvements over the raw SAM3. The comparisons with conventional LoRA and existing SAM3 calibration methods support the effectiveness of the specific design. In overall, the empirical results are strong and convincing.

Critical weaknesses

1. Limited technical novelty of individual components and the positioning of the main contributions

This paper's main contribution is the SAM3-specific architectural design and integration of existing techniques, rather than fundamentally a new mechanism. For example, image-conditioned text adaptation has been explored in CoCoOp, while the proposed SBR and CWD are closely related to existing pairwise ranking and confidence-weighted pseudo-label learning techniques. In addition, recent concurrent works, such as ConceptBank and CoCo-SAM3, have also discussed related concept recognition issues in SAM3. These observations do not make this paper's novelty meaningless, especially considering its strong empirical performance and direct comparisons with these methods. However, the authors should more clearly distinguish the novelty of their specific ARP design. A more precise positioning would better clarify the actual contribution of this work.

2. Insufficient evidence to fully support the claimed recognition bottleneck in SAM3 and the improvement mechanism

The paper provides interesting oracle experiments to demonstrate that concept recognition is the primary bottleneck of SAM3. However, the GT category correction in Figure 2 relies on ground-truth information to relabel predicted regions, and the spatial quality analysis in Appendix D.5 mainly considers successfully recognized objects. Whether the masks of misclassified or missed objects are also sufficiently accurate remains unclear. In addition, ARP modifies not only the presence predictions but also the text-conditioned fusion and decoder paths, which may change the predicted mask geometry. Although the presence-only comparisons in Figure 7 are useful, the current experiments do not fully separate the improvements in concept recognition from those in spatial prediction. These limitations weaken the evidence of the proposed mechanism.

In overall, I think the proposed SAM3-specific adaptation framework is meaningful, and the experimental results are relatively strong. Although the technical components have been discussed in prior and concurrent works, and the main mechanism could be better validated, the performance improvements make the paper a valuable contribution. I currently lean to a weak acceptance, and I encourage the authors to better position their novelty claims and better clarify the underlying improvement mechanism.

Questions for authors

1 . The spatial quality analysis in Appendix D.5 mainly evaluates successfully recognized objects. Could the authors also report the class-agnostic mask IoU or boundary quality for misclassified and missed objects, using the raw SAM3 mask predictions? This would provide more direct evidence that concept recognition rather than mask quality is the main bottleneck.

2 . Since ARP modifies the text-conditioned fusion and decoder paths, the improvements may also come from the changes in mask geometry. Could the authors disentangle these contributions, for example, by evaluating TAG's presence scores with frozen SAM3 masks and frozen SAM3's presence scores with TAG masks? This would help clarify if the improvements are actually driven by semantic calibration as claimed.

1 comment

Thanks for the detailed response and additional experiments. I remain my positive recommendation for the paper's acceptance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.