- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TAG adapts a frozen SAM3 to downstream universal segmentation (semantic, instance, panoptic) using only image-level class labels. Oracle interventions are used to argue that SAM3's downstream failures are concept-recognition errors rather than mask errors. A 3.1M-parameter Anchor Residual Pathway (low-rank and image-conditioned residuals on text features, fusion-encoder text K/V, decoder image K/V, and the presence token) is trained with BCE on presence plus a pairwise hard-negative ranking loss. TAG-V replaces human labels with Qwen3.8-27B presence probabilities through confidence-weighted distillation. On ADE20K, Cityscapes, and iSAID, TAG-I improves frozen SAM3 by +17.2 PQ, +9.5 AP, and +20.8 mIoU on average (3 seeds, one shared configuration); TAG-V by +14.0, +7.5, and +16.8. Appendices add VOC and COCO results, comparisons with training-free SAM3 calibration methods, LoRA and presence-classifier controls, and hyperparameter sweeps.
Critical strengths
-
Large, consistent gains at low cost. With the backbone frozen, one shared hyperparameter set, and 3-seed averaging, TAG improves frozen SAM3 by 12 to 25 PQ across natural, urban, and aerial domains. This is a practically useful recipe for deploying SAM3 on a closed vocabulary without masks.
-
Controls isolate where the gain comes from. Output-level presence calibration (class-wise affine, image-based and token-based classifiers) and a budget-matched LoRA are all clearly below TAG-I, showing that adapting how text conditions the model matters more than re-scoring presence on fixed features.
Critical weaknesses
-
The baselines that most directly threaten the contribution are missing. For TAG-V, gating frozen SAM3 at test time with the same Qwen3.8-27B presence predictions (or SAM 3 Agent) is the natural reference; if it matches TAG-V, the contribution reduces to efficiency. For TAG-I, prompt or class-embedding learning on a frozen VLM with the same BCE plus ranking objective (CoOp/CoCoOp style; Parisot et al., CVPR 2023, learn class-name embeddings on a frozen model for exactly this purpose) is the natural reference; the budget-matched LoRA adapts only cross-attention K/V and is not a substitute. Without these, it is unclear whether ARP's specific design matters or whether any text-side adaptation with this objective would do.
-
Claims outrun the evidence. The "GT categories" oracle assigns each predicted component its majority GT label, which injects GT spatial information, and the "GT mask regions" oracle scores below frozen SAM3 (22.3 vs. 37.5 mIoU) without explanation. The natural ceiling for image-level supervision, keeping GT-present classes and suppressing absent ones, is not reported, so the "recognition, not masks" diagnosis is asserted more strongly than shown. On VOC and COCO, frozen SAM3 alone already beats every weakly supervised baseline (VOC 83.5 vs. 82.6 mIoU; COCO 43.9 vs. 25.2 AP), yet App. D.2.3 attributes the full margins to TAG. SAM3 is also pretrained on masks from adjacent domains (BDD100K and VisDrone are in SA-Co/EXT), so "without dense supervision" holds only for the target dataset, and "surpassing fully supervised methods" is true for two selected cells.
Questions for authors
-
How does frozen SAM3 perform when gated at inference by the same MLLM presence predictions used to train TAG-V?
-
How does a CoOp/CoCoOp-style or Parisot-style class-embedding baseline perform under your BCE plus SBR objective?
-
How is a category assigned to each region in the "GT mask regions" oracle, and what does the full GT-presence oracle (present classes set to 1, absent classes set to 0) score?
-
What are the latency and memory per image on ADE20K, given per-class fusion and decoder passes over 150 classes?
Thanks for the detailed response and additional experiments. I remain my positive recommendation for the paper's acceptance.