- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 1
Read review
Paper summary
The authors propose a new finetuning method for visual grounding. They try to use the attention distributions of the [SEG] token from the multiple layers and heads in VLM as the prompt for decoder for mask prediction.
Critical strengths
The proposed method achieves substantial performance improvements over existing baselines, demonstrating its potential effectiveness. Moreover, the proposed method is relatively straightforward. Since existing methods tend to produce high-confidence false-positive predictions, the authors introduce negative categories during training to suppress these incorrect predictions.
Critical weaknesses
The writing quality is poor, making it difficult to follow the motivation and methodology. I am not confident that I have fully understood the proposed approach. Furthermore, the training details do not clearly specify whether novel categories are included during training as it needs to use VLM to expand the absolutely absent categories. If so, the performance comparison would be unfair, as it would violate the fundamental principle of open-vocabulary detection (OVD), which aims to detect categories unseen during training. Furthermore, the selection of the 40 absolutely absent categories for each sample requires further explanation. The authors should clarify how these categories are selected and whether they include novel categories reserved for testing.
Questions for authors
- What is the motivation. I can just guess about that the authors believe single [SEG] token cannot encode spatial information. However, the spatial information is important for segmentation prediction. Then, the authors find that the attention map of the [SEG] token can represent the spatial information since the high values represents the [SEG] token are more related to that image token. Thus, the authors try to use the attention map of the [SEG] token as a prompt for decoder to predict the segmentation mask.
- For figure 2, could you please explain how to derive the different diagrams with detailed implementations. Did you just use the row attention of [SEG] token to replace the original [SEG] token in the baseline?
- In my opinion, the [SEG] token may encode certain spatial information, as each image token is associated with positional encoding. Therefore, it would be helpful for the authors to clarify how the proposed method distinguishes or extracts spatial information from the [SEG] token.
- The equations lack sufficient definitions of the mathematical notations used. The authors should clearly define all variables and symbols to improve readability and ensure that readers can fully understand the proposed method. For example, there is no explanation on S in line 240. What is w_k^{(0}} in Eq. (2). What is c and the symbol of a circle with center dot in Eq. (3)
- a is an attention map, m is the mask. How to realize the cosine similarity in Eq. (3).
- The network output appears to be an attention map, which is typically normalized using a softmax function, resulting in values between 0 and 1. However, unlike a ground-truth (GT) binary mask, which contains multiple pixels with values of 1 and 0, a softmax-normalized attention map represents a probability distribution, where the values are generally continuous and sum to 1 along the normalized dimension. Therefore, it is unclear how the proposed method addresses the discrepancy between the prediction and the GT mask in terms of their value distributions. The authors should clarify how the attention map is converted into a segmentation prediction and how the loss function is designed to accommodate this difference.
6 comments
Authors Response 1, Part 2
Q2) Are novel or "absent" categories used in training?
No. AttnSeg is a referring and reasoning segmentation method, not an open-vocabulary detector, and it uses no category vocabulary. Each training sample is a triplet: an image, a free-form expression, and the ground-truth mask of the region it refers to. There is no list of absent, novel or negative categories, so no category selection is involved. We train on the thirteen-dataset mixture released with Sa2VA (Appendix D, Table 6). We evaluate on ReasonSeg and GS-Eval, whose training data is not in that mixture; our ReasonSeg results use no ReasonSeg training data at all.
False positives are suppressed by an emergent property, not by negative categories. Figure 3 shows the learned head weights on a real example, "the door with number 8":
- The convolution gives positive weights to heads on the correct door and on the number 8.
- It gives negative weights to a head locked onto the other door (a distractor), to a head collapsed onto an attention sink, and to a diffuse head. These heads are therefore subtracted from the map.
Nothing supervises these roles. The convolution is trained only to make the combined map match the ground-truth mask, as in any segmentation task, and the ability to suppress distractors emerges from that objective. This is the VLM doing the heavy lifting: its attention becomes discriminative and separates the referent from look-alikes before the decoder sees it. Across the trained model, about 40% of the weights are negative (Sec. 3.3).
Q3) How are the diagrams in Figure 2 derived, and does AttnSeg simply replace the [SEG] token with its attention row?
Figure 2 is our probing analysis (Sec. 3.2), on a frozen Sa2VA-4B. Each point is a small readout trained to predict the mask from one internal signal of the frozen model, scored in gIoU:
- Panel (a) compares two signals, the [SEG] token's attention maps (purple) and its hidden state (green). Each is read by three readouts of increasing capacity: a single convolution, a U-Net, and a U-Net that also sees the RGB image. The two signals differ only in the input; the readout and its training are identical (Appendix B).
- Panel (b) repeats the probe on each window of five consecutive layers, to find where in the network the localization lives.
The per-head maps and their learned weights are shown in Figure 3 (Q2).
The replacement is not arbitrary; it is what the analysis prescribes. Figure 2 shows three things:
- the attention carries 18 to 22 gIoU more localization than the hidden state (Finding 1);
- it already encodes the referent's extent (Finding 2);
- it lives in the late layers (Finding 3).
AttnSeg therefore prompts the decoder with the [SEG] token's attention instead of its hidden state. This interface is the first of three components. The other two, supervising the attention with and the two-stage training, make the VLM do the heavy lifting of localization. Our ablation (Table 3) shows that the gain needs all three. The interface alone adds +0.26 / +1.09 gIoU over the same model finetuned on the same data, while the full method adds +4.19 / +9.56.
Authors Response 1, Part 3
Q4) Doesn't the [SEG] token already encode spatial information, and how does AttnSeg extract it?
The [SEG] hidden state does carry some spatial information, and measuring how much is exactly what our analysis does. We use probing (Sec. 3.2): we freeze the model, train a small readout to predict the mask from one internal signal, and take the readout's accuracy as a measure of what that signal already contains. The hidden state gives 30.6 / 35.0 gIoU (ReasonSeg / GS-Eval). Its spatial information is real but limited: it is a single vector of 2048 numbers with no spatial index, so it must be decoded back onto the image.
AttnSeg instead reads the attention weights that the [SEG] token assigns to the image tokens. This row has one entry per image token, so it is indexed by image position by construction and folds directly onto the grid, with nothing to decode (Eq. 2). Read by the same readout, it gives 52.1 / 53.2, 18 to 22 gIoU more than the hidden state (Finding 1). The localization concentrates in the late layers (Finding 3). The learned combination (Eq. 3) selects and weights the heads automatically, so no per-model analysis is needed.
Q5) Can the notation be defined?
The revision defines every symbol where it is introduced (Sec. 3.1 to 3.3), and Figure 3 annotates the shape of every tensor. The symbols the reviewer lists are:
- (Eq. 2): the length of the token sequence, which comprises the image tokens, the text tokens and the generated answer. The attention weights of one forward pass form a tensor of shape , for layers and heads.
- (Eq. 3): the initial value of the learned weight of layer and head . It is set to the uniform average , so training starts from the plain mean of the heads.
- (Eq. 4): per-patch weights, on patches inside the ground-truth mask and elsewhere, which up-weight the foreground.
- and the cosine (Eq. 4): is the elementwise product. Let be the combined attention map and the ground-truth mask on the same grid, both flattened to vectors. The weighted inner product is , the weighted norm is , and the loss is . In code, this is an elementwise product with followed by sums.
Q6) How is a softmax attention map compared with a binary mask, and how does it become a segmentation?
This difference is exactly what our objective is designed for. As Sec. 3.3 states, "since only where the map attends carries information, and not how much attention reaches the image, any such objective must be invariant to the scale of ." The method handles it in three places:
- Normalization (Eq. 2). Each head's map is rescaled to spatial mean one. A head is then characterized by where it attends, not by how much of its mass reaches the image, which varies with the sequence length.
- A scale-invariant loss (Eq. 4). The cosine compares the shape of the map with that of the mask, not their values, so the map does not need to be binary or to sum to one. After the signed combination, the map is no longer a probability distribution anyway. The loss is minimized when the map is uniform over the target and flat elsewhere. This is why we chose a cosine rather than a loss that compares values, such as binary cross-entropy (Sec. 3.3).
- The decoder, not the map, makes the mask. The combined map is min-max normalized, resized and converted to logits (Appendix C, Eq. 7). It then enters SAM2's dense mask-prompt channel without thresholding. SAM2, trained with the standard mask loss in stage two, outputs the binary mask at full resolution.
We hope this clarifies the approach. The rewritten paper and the animation (https://anonymous00440055.github.io/attseg_project_page/) walk through it step by step, and we would be glad to answer any further question.
References:
[1] Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videos
[2] SAM 2: Segment Anything in Images and Videos
Authors Response 1, Part 1
We thank the reviewer for their time and for recognizing the substantial gains over existing baselines and the simplicity of the method. We are grateful for this rebuttal, as it gives us the chance to correct a misunderstanding of the paper's premise. AttnSeg is not an open-vocabulary detection method. It uses no category vocabulary: no novel or held-out categories, no "absolutely absent" categories, and no negative categories during training. AttnSeg is a referring and reasoning segmentation method that changes how a vision-language model (VLM) passes the location of the referent to its mask decoder. Instead of a single [SEG] vector, AttnSeg passes the [SEG] token's own attention map, and trains the VLM so that this map localizes the referent. The VLM thus does the heavy lifting of localization, and the decoder only refines. Q1 explains this step by step. We have also rewritten large parts of the paper, the method section in particular, and we provide an anonymous page with an animation of the method: https://anonymous00440055.github.io/attseg_project_page/.
Q1) What is AttnSeg, and what motivates it?
We restate the core of the paper here, from the question it asks to the method that answers it.
The research question. Segmentation VLMs such as Sa2VA [1] mark the referent with a [SEG] token and pass the token's final hidden state, a single vector, to a mask decoder such as SAM2 [2]. The decoder never sees the text, so this vector must carry both where the referent is and how much of it is meant: "the left ear of the dog" must be told apart from "the dog". We ask two questions:
The analysis behind each premise (Sec. 3.2). We test these premises with probing, a standard way to measure what a frozen network already knows. We freeze the model and train a small readout to predict the mask from one of its internal signals; if the small readout succeeds, the signal already contains the mask. On a frozen Sa2VA-4B:
AttnSeg in four steps, with the tensor shapes for Sa2VA-4B (L=36 layers, H=16 heads, S tokens in the sequence, an h×w grid of image tokens):
This gives +6.4 / +10.2 gIoU at 4B and +8.8 / +11.3 at 14B. The largest gains are on object parts, multiple objects and single objects, where the VLM must single out the referent.