acceptodds
Withdrawn from ICLR 2027

BTGNet: A Background Text Guided Model with Joint Query Refinement for Few-Shot Semantic Segmentation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing few-shot semantic segmentation (FSS) methods have extensively explored visual background information to suppress distracting regions, while multimodal approaches mainly rely on foreground-oriented text prompts. In contrast, the potential of background words as an explicit textual prior remains largely unexplored. We observe that category-related background words can provide complementary language cues for distinguishing target objects from visually confusing regions, while being easier to obtain and extend than image-specific background annotations. Based on this insight, we propose BTGNet, a Background Text Guided Network that explicitly incorporates background words into few-shot segmentation. Specifically, we employ a large language model to automatically generate task-adaptive, category-related background vocabularies and align them with visual representations through a frozen CLIP encoder to guide Grad-CAM-based query activation. To further improve the coarse activations produced by frozen CLIP, we propose a joint refinement strategy that combines fixed attention priors with learnable dynamic refinement. Extensive experiments on PASCAL-5\(^i\) and COCO-20\(^i\) demonstrate consistent improvements over state-of-the-art methods. Further analyses show that semantically relevant background words, rather than arbitrary additional text, are critical to performance gains, highlighting background words as a generalizable textual prior for multimodal FSS.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Few-Shot Semantic Segmentation, Background Text Guidance, Multimodal, CLIP, Query Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NjQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BTGNet for few-shot semantic segmentation. It uses an LLM to generate category-specific background words, incorporates them into CLIP-based Grad-CAM generation, and refines the resulting activations through fixed and learnable branches.

Critical strengths
  1. The motivation is reasonable. Foreground-only activation can include co-occurring background regions, and explicitly representing competing background concepts can help distinguish the target from contextual distractors. This is particularly relevant when only a few annotated support images are available.
  2. The ablations provide useful evidence for the proposed components. The paper separately examines background-word guidance and activation refinement, and compares random, fixed, and LLM-generated vocabularies. Prompt and cross-LLM analyses further examine the stability of vocabulary generation.
  3. The counterfactual background evaluation is a valuable addition. Preserving the foreground while replacing the query background directly examines whether the method depends on typical object-context associations. The reported results suggest that the benefit is not explained solely by background co-occurrence shortcuts.
Critical weaknesses

1.The methodological novelty needs clearer positioning relative to CLIP-ES (https://arxiv.org/abs/2212.09506) and WeCLIP (https://arxiv.org/abs/2406.11189). CLIP-ES already incorporates background text into softmax-based Grad-CAM and attention-based refinement. WeCLIP further uses learned feature affinity to select frozen CLIP attention maps through an average-distance threshold, which is closely related to BTGNet’s dynamic refinement. Although the task settings and implementations differ, the paper should explain more precisely what advances beyond category-specific vocabulary generation and the combination of existing refinement principles. 2. The attribution of performance gains across BTGNet-P/H/D remains insufficiently controlled. Evaluating several integrations is useful, and Appendix Table 14 provides additional evidence of applicability. However, the listed pretrained representations differ between the original baselines and some BTGNet variants, making it difficult to attribute the reported gains specifically to background-word guidance. The component ablation on BTGNet-H supports the design within that configuration, but does not establish whether the same components explain the gains in BTGNet-P and BTGNet-D. 3. The published vocabularies contain apparent foreground-background conflicts. Table 12 includes the target itself in several background lists. Since these terms participate in the softmax competition used for activation generation, the authors should clarify whether they were used in the experiments or are errors in the table, and assess their effect.

Questions for authors
  1. Table 5 compares fixed and LLM-generated vocabularies, while Table 3 examines refinement components. Could the authors cross these factors by evaluating both vocabulary types with and without joint refinement, using the same FSS configuration and controlling vocabulary size? Please also clarify whether the fixed lists correspond to those used in CLIP-ES. This would separate vocabulary improvements from refinement benefits.
  2. Could the authors clarify the methodological differences from WeCLIP and compare its refinement module with BTGNet’s joint refinement within the same FSS pipeline, using identical initial CAMs and background vocabulary?
  3. Are the baseline-to-BTGNet comparisons for P/H/D matched in both the segmentation backbone and pretrained representation? Where they are not, could the authors provide matched baselines? A compact component ablation for P and D would also help establish whether the contributions observed on H carry over to the other integrations.
  4. Were the foreground terms listed as background candidates in Table 12 actually used? Please clarify the filtering procedure and, if they were used, evaluate the effect of removing target names and synonyms.
Reviewer_29D1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BTGNet for few-shot semantic segmentation. The method uses an LLM to generate category-specific background words and incorporates these descriptions into CLIP-based Grad-CAM activation generation. A fixed refinement branch and a learnable dynamic branch further refine the activations before they are integrated into an existing segmentation framework. Experiments on PASCAL-5i and COCO-20i report improvements across several base models. Additional experiments examine background vocabulary selection, prompt variation, different LLMs, and changes in background context.

Critical strengths
  1. The paper examines a useful role for background text in few-shot segmentation. The comparisons in Table 5 help distinguish the benefit of meaningful background descriptions from simply adding more text. On COCO-20i, LGBW achieves 62.1 mIoU, compared with 60.8 for the COCO-specific fixed vocabulary and 56.0 for random background words. These results provide relevant evidence for using category-related background semantics.
  2. The evaluation covers both applicability across models and sensitivity to background context. Table 14 reports paired comparisons within several segmentation frameworks, which supports the usefulness of the proposed strategy beyond a single base model. Table 15 also considers counterfactual background replacements. The full model maintains 79.6 mIoU under atypical backgrounds, compared with 80.5 under natural backgrounds. This directly addresses a concern closely tied to the motivation: whether category-specific background descriptions create excessive dependence on typical visual contexts.
Critical weaknesses
  1. The foreground-only control and the refinement ablations need clearer specification. Equations (7)–(8) compute the foreground log-probability after a softmax over foreground and background text similarities. If the no-background setting simply removes all background entries, the foreground probability becomes one and its gradient becomes zero. The paper should explain the scoring function actually used in this setting. This is an ambiguity in the description rather than evidence that the implementation uses a degenerate control.

    Table 3 also does not report a variant with both DR and FR enabled but LGBW disabled. The “w/o LGBW” result of 74.7 in Table 15 matches the FR-only result in Table 3, so it would be helpful to confirm whether both refinement branches remain enabled. This clarification matters for attributing the improvement specifically to background words.

  2. Some implementation details and reported results need to be reconciled. Sections 3.2–3.4 describe intermediate features and attention matrices from a 12-layer Transformer, whereas BTGNet-P and BTGNet-H are listed with CLIP-RN50. The paper should explain how the corresponding features and attention matrices are obtained for these versions. Section 3.4 also refers to support-image attention matrices when refining query activations. It is unclear whether this is a notation error or an intentional operation requiring spatial correspondence.

    In addition, Table 2 reports 62.1 mIoU for BTGNet-H on COCO-20i in the 1-shot setting, while Table 16 reports 58.1 on the RTX 4090. The latter table does not clearly specify the shot setting or explain a configuration change. These entries should be checked and their experimental settings stated explicitly.

Questions for authors
  1. What scoring function is used to generate Grad-CAM maps in the no-background setting? Does the “w/o LGBW” variant in Table 15 retain both DR and FR?
  2. How are the intermediate features and attention matrices constructed for the CLIP-RN50 versions? In Section 3.4, should the attention matrices come from the query image, or are support-image matrices intentionally used? If the latter, how is spatial correspondence established?
  3. What explains the BTGNet-H COCO-20i results of 62.1 in Table 2 and 58.1 in Table 16? Please specify the shot setting and any differences in model configuration or evaluation protocol.
Reviewer_gSd4OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes BTGNet, a few-shot segmentation (FSS) method that uses text to guide query activation and suppress irrelevant regions. An LLM generates class-specific background words, which are used alongside the class name in CLIP-based Grad-CAM to highlight the target and suppress the background. The resulting activation map is refined through a fixed attention-based branch and a learnable branch, and the refined maps are fed into an existing FSS model. The paper report state-of-the-art results on PASCAL-5ᶦ and COCO-20ᶦ.

Critical strengths

The paper presents an intuitive idea, using background words to help distinguish the target object from surrounding regions. The method can be added to existing few-shot segmentation models and reports improvements over several base models on two standard datasets. The authors include a broad set of analyses covering different prompts, language models and background conditions. Generating and storing the background words once per class avoids repeated LLM calls and adds no trainable parameters or computation at inference.

Critical weaknesses

The major weakness is novelty of the proposed method. CLIP-ES [1] already uses background words with softmax Grad-CAM to reduce background confusion, although in weakly supervised segmentation. PI-CLIP [2], which the paper acknowledges, also uses the same fixed refinement procedure based on attention averaging, Sinkhorn normalization, and higher-order attention. LLM-generated class-specific text for CLIP is also established (CuPL [3]). CLIMS [4] uses class-related background text, to suppress co-occurring context. The paper should more clearly establish the novelty of its specific combination and the contribution of its added components.

It seems that under Eqs. 7- 8, removing all background words gives zero gradients, and the stated refinement cannot make a zero map nonzero. Table 3 therefore suggests an undocumented implementation difference or reporting inconsistency. Until clarified, the ablation does not cleanly isolate the benefit of background words.

The auxiliary loss weight is tested only at the chosen value and above, without evaluating smaller values or removing the loss while keeping the architecture unchanged. The pseudo-label threshold is not specified, and sensitivity to background-vocabulary size and CLIP layer selection is not systematically evaluated. Efficiency results cover only BTGNet-H. The authors should clarify whether the reported forward FLOPs include fixed refinement matrix operations, report the additional cost of Grad-CAM’s test-time backward pass, and confirm that this pass is included in the reported end-to-end FPS.

The experiments do not adequately separate the proposed improvements from other changes. BTGNet uses 200 training epochs, while the original PI-CLIP reports 30, making a comparison under the same training budget necessary.

[1] CLIP is Also an Efficient Segmenter: A Text-Driven Approach for Weakly Supervised Semantic Segmentation, Lin et al., CVPR 2023

[2] Rethinking Prior Information Generation with CLIP for Few-Shot Segmentation, Wang et al., CVPR 2024

[3] What does a platypus look like? Generating customized prompts for zero-shot image classification, Pratt et al., ICCV 2023

[4] CLIMS: Cross Language Image Matching for Weakly Supervised Semantic Segmentation, Xie et al., CVPR 2022

Questions for authors
  • The main table gives BTGNet-H a COCO score of 62.1, while the hardware comparison reports approximately 58.1 without explaining the difference.

  • BTGNet adds a frozen CLIP, an LLM and test-time class names on top of each base model, yet Table 1 lists only “CLIP-RN50” as the backbone. Most of the gain over the base models likely comes from adding CLIP priors, which PI-CLIP already does. The paper’s own CLIP-prior configuration without background words (74.7) scores below PI-CLIP’s published 76.8, which inflates the apparent benefit of background words.

  • please clarify whether RN50 denotes the segmentation backbone or the CLIP image encoder. Both refinement branches are described using support-image attention to refine query activations, without explaining spatial alignment.

  • For each fold, do the base-class background vocabularies contain held-out novel-class names or their synonyms during training? How does performance change when these terms are filtered out?

  • Please plot the per-class mIoU gain from LGBW against the number of background words and explain the particularly large 12.6-point improvement on PASCAL-5ᶦ fold 3 (63.8 → 76.4 in Table 3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.