acceptodds
Under review as a conference paper at ICLR 2027

SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models

Abstract

Recent works show that Vision-Language Models (VLMs) like CLIP struggle with logical reasoning, which can contribute to unreliable model behavior and hallucinations. An example of such logical operators is negation. Given a prompt like “retrieve (or generate) a street scene without pedestrians”, VLMs often fail to respect the “not”. Existing methods try to address the logical blind spots of VLMs by fine-tuning models on larger and richer datasets. However, this fine-tuning can compromise zero-shot generalization and often fails to fully eliminate these blind spots. In this paper, we use the negation operator as a case study and show that representing logical operators as traditional point embeddings in a joint embedding space can be limiting. Instead, we propose subspace modeling of negation, where text representation is a subspace in the embedding space rather than a traditional vector (point) embedding. Then we derive a closed-form Mahalanobis distance on a curved surface from an image embedding (point) to a text representation (subspace), replacing the conventional cosine similarity between image and text embeddings. Across retrieval and multiple-choice-question (MCQ) tasks, our new modeling improves negation understanding in VLMs including CLIP, SigLIP, and AIMV2 by about 30% on average over the baseline, without any additional re-training of the VLM, while preserving the model's zero-shot performance. More broadly, our framework could serve as a starting point for research on subspace modeling of other logical operators, not only at inference time, but also during model pre-training. Code is included in the supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Representation Learning, Subspace Modeling, Vision-Language Models (VLMs), Logical Reasoning, Negation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hHVyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the negation understanding problem in Vision-Language Models (VLMs), e.g., retrieving a picture of a dog not on grass. Specifically, it proposes a test-time steering method that uses Mahalanobis distance in the subspace to produce a refined score. However, during inference, it always requires a Large Language Model (LLM) to parse the caption, which is impractical in high-throughput VLM deployments. In addition, this paper lacks ablation studies demonstrating that the Mahalanobis distance outperforms other distance metrics or scoring functions, failing to verify its core contribution.

Critical strengths
  1. This paper studies a very practical problem in VLMs, i.e., negation understanding, which could benefit a wide range of real-world applications.
  2. The paper is well written and easy to follow.
Critical weaknesses
  1. The main limitation is the practicality of the proposed method. During inference, it always requires an LLM (e.g., Mistral-7B) to parse the caption, and the resulting latency and computational cost undoubtedly limit high-throughput VLM deployments. For example, in image-to-text retrieval, a VLM may need to efficiently score thousands of candidate captions.
  2. The paper lacks a comparison with test-time adaptation methods [1] and post-hoc correction methods [2]. In addition, the related work section would benefit from a broader discussion of negation understanding beyond embedding VLMs, e.g., including applications in MLLMs and generation models.
  3. One core contribution of this work is to compute the matching score using the Mahalanobis distance. However, the paper lacks ablation studies demonstrating the superiority and necessity of this metric.
  4. The T2I generation results suggest that \hat{d} can serve as an effective point representation for a negated caption. This appears to be in tension with the claim that point embeddings in a joint embedding space can be limiting.

[1] Negation-aware test-time adaptation for vision-language models. TPAMI 2026.

[2] What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features. ECCV 2026.

Questions for authors
  1. The paper states that the results of Aggarwal et al. (2026) could not be reproduced. Please clarify whether this was based on an independent reimplementation or on running the official code [1] released by the authors.

[1] Implementation of the Paper "Seeing What's Not There: Negation Understanding Needs More Than Training". ICLR 2026. URL https://github.com/ananta-nirvana-ml/negation-understanding

7 comments
AuthorsOpenReview ↗

Addressing Weaknesses and Questions for Reviewer hHVy (Part 1/6)

(Part 1/6)

We sincerely thank Reviewer hHVy for their detailed and thoughtful review. Below, we address their concerns one by one.

Weakness 1

As in our response to Reviewer gxYg, we emphasize that our paper's contribution is modeling negation as a subspace in the embedding space, once its affirmative and negated parts are available. For evaluation, we need to handle natural negation forms in NegBench, such as implicit negation, and we use an LLM for this, since rule-based methods cannot easily identify them. However, any other method can be used instead, including a rule-based one for simpler negated captions, to improve speed in practice: the splitter is modular and can be replaced by any other approach. So, we do not always need an LLM at inference time. In this paper, we used LLMs to keep the evaluation simple, since our contribution lies elsewhere, but practical applications can use a smaller LLM, a rule-based method, or a hybrid of the two for better throughput and efficiency. Moreover, as a side analysis of caption splitting, Fig. 5 shows that a 7B model is not necessary: 1B models give the same performance with much lower latency, and a 360M LLM (of the same order of magnitude as CLIP itself) gives slightly lower performance but is still a much better choice than vanilla CLIP.

Weakness 2

We thank the reviewer for pointing us to these recent post-hoc works on negation. Both are concurrent with our submission: [1] was published in TPAMI in October 2026, after we submitted our work to ICLR 2027, and, since we compare only with peer-reviewed methods, [2] became available for comparison only when the ECCV 2026 proceedings were released in September 2026, close to the ICLR deadline. It was therefore not feasible for us to compare with them in the submitted version. Below, we discuss both works and how their methodology differs from ours.

NEAT [1] (Han et al., TPAMI 2026) also targets negation in CLIP-like models without large-scale post-training, but its methodology differs from ours in three main ways. First, NEAT is a test-time adaptation method. It attributes the failure to a dual-concept distribution shift between affirmative and negated captions, and it corrects this shift by updating the LayerNorm parameters of the text encoder with gradient descent on batches of unlabeled test data. SpaceVLM, in contrast, updates no parameters and computes no gradients (with a single global t, which we show to work well in our response to Reviewer gxYg). It scores each query independently, so it needs no test-time batch, optimizer, or learning rate. Second, after adaptation, NEAT still represents each caption as a single point embedding, and the shift it corrects is defined over distributions of such points. SpaceVLM instead represents a negated caption as a region on the unit sphere, namely the intersection of an affirmative cap with the complement of a negated cap, and scores images by a closed-form anisotropic geodesic distance to this region. Third, like SpaceVLM, NEAT uses an LLM to split a negated caption into its affirmative and negated parts, but the two methods use these parts differently. NEAT combines them with a heuristic product rule, (t^P·v)(1 − α[t^N·v]₊), with a fixed trade-off α, and uses this rule only to select pseudo-targets for its losses; it also needs an LLM-generated reversed caption. In addition, because SpaceVLM leaves both encoders unchanged, its behavior on affirmative captions is unchanged by construction, whereas NEAT reports changes of −4.75% to +4.91% in standard retrieval. Finally, on COCO MCQ with CLIP ViT-B/32, NEAT improves accuracy from 39.25% to 48.41%, while SpaceVLM improves it from 39.2% to 66.7% (Table 2).

(Continued in Part 2/6.)

AuthorsOpenReview ↗

Addressing Weaknesses and Questions for Reviewer hHVy (Part 2/6)

(Part 2/6)

PeakPatch [2] (Lu et al., ECCV 2026) also keeps the VLM frozen, but it differs from SpaceVLM in what it assumes and in what it needs. PeakPatch locates the loss of negation inside the text encoder: negation is separable in intermediate layers but collapses in the final layers. It therefore trains two modules (5.2M parameters) on about 1.06M CC12M caption pairs and 313K synthetic MCQ samples with LLM-generated negations: an Embedding Correction Network, which reads intermediate-layer tokens and adds a learned offset to the final text embedding, and a Score Correction Network, which adds a learned, bounded offset to the image–text score for pairwise tasks such as MCQ. SpaceVLM, in contrast, uses only the final normalized embeddings. It needs no training data, no added parameters, and no access to intermediate activations, and it derives its scoring rule in closed form, with a single global threshold t shared across tasks. Because affirmative captions reduce to vanilla CLIP scoring in SpaceVLM, its behavior on them is unchanged by construction, whereas PeakPatch alters every text embedding (e.g., −2.8 pp zero-shot accuracy on ImageNet). The two methods also represent negation differently. The corrected PeakPatch embedding is still a single point, and the authors note that, because negation is set-valued, their method recovers the affirmative/negated distinction but does not resolve a negated phrase to its complement; their own visualization shows negated captions spreading into a looser shared region rather than collapsing to a point. SpaceVLM models this region explicitly, as the intersection of an affirmative cap with the complement of a negated cap, and scores images by their distance to it. In exchange, PeakPatch needs no caption splitter at inference, whereas SpaceVLM relies on one; our splitter is modular, and Fig. 5 and Table 4 show that a 1B LLM or manual splitting yields similar gains. Empirically, the two approaches are complementary: with CLIP ViT-B/32, PeakPatch is stronger on COCO MCQ (74.3% vs. 66.7%) and on COCO negated retrieval, while SpaceVLM is stronger on VOC2007 MCQ (80.9% vs. 65.5%) and on text-to-image generation (combined accuracy 59.7% vs. 45.8%), without any training.

(Continued in Part 3/6.)

AuthorsOpenReview ↗

Addressing Weaknesses and Questions for Reviewer hHVy (Part 3/6)

(Part 3/6)

Regarding the broader related work, and in line with our response to Reviewer 1SWE, we will add the following discussion to the related work section:

Region or distribution embeddings in VLMs. PCME (Chun et al., CVPR 2021) and ProbVLM (Upadhyay et al., ICCV 2023) represent an image or a caption as a probability distribution instead of a point. MERU (Desai et al., ICML 2023) and HyCoCLIP (Pal et al., ICLR 2025) represent a text as an entailment cone in hyperbolic space. These works target ambiguity (one-to-many matching) or hierarchy, not logical operators such as negation, and all of them need training (a new model, or an additional adapter in ProbVLM).

Negation as a region outside VLMs. Widdows (ACL 2003) models negation in text retrieval as a projection onto an orthogonal subspace, and BetaE (Ren & Leskovec, NeurIPS 2020) models negation in knowledge-graph queries with Beta distributions, because points and boxes cannot represent it. These works address text-only or knowledge-graph embeddings. The first uses flat linear subspaces, not the spherical geometry of VLMs, and the second learns the embeddings and the logical operators from scratch. Negation is also an open problem for generative MLLMs. NegVQA (Zhang et al., ACL 2025) tests 20 VLMs from seven model families, including LLaVA and Qwen2-VL, and reports a substantial accuracy drop on negated questions compared with the original questions.

Weakness 3

As in our response to Reviewer gxYg, we provide the full ablation study here.

To avoid confusion with the tables in the paper, we number the tables in this response R1–R4.

Table R1 — Negated retrieval, R@1 / R@5 / R@10 (Neg). C1 and C2 use t = 0.92.

DatasetModelAffirmative queryA. Affirmative onlyB. eₐ − λ·eₙλC1. Single embedding −d̂C2. Mahalanobis −D
COCOCLIP29.8 / 54.8 / 66.029.7 / 54.8 / 66.230.9 / 56.2 / 67.50.2529.8 / 54.9 / 66.229.8 / 54.9 / 66.2
SigLIP-248.1 / 72.6 / 80.947.8 / 72.8 / 81.148.3 / 73.4 / 81.50.2547.9 / 72.9 / 81.347.8 / 72.8 / 81.2
AIMV241.5 / 66.8 / 76.541.4 / 67.0 / 76.541.7 / 67.3 / 76.80.141.5 / 67.1 / 76.641.5 / 67.1 / 76.6
MSR-VTTCLIP28.5 / 49.6 / 61.128.9 / 52.2 / 62.428.8 / 52.4 / 62.60.0528.8 / 52.1 / 62.828.6 / 52.1 / 62.8
SigLIP-235.2 / 59.0 / 70.930.6 / 54.3 / 63.431.0 / 54.7 / 63.70.0530.3 / 54.8 / 63.630.4 / 54.2 / 64.0
AIMV231.3 / 55.9 / 65.531.3 / 56.5 / 66.631.3 / 56.7 / 66.20.131.6 / 56.7 / 65.831.5 / 56.7 / 65.8
Mean35.7 / 59.8 / 70.235.0 / 59.6 / 69.435.3 / 60.1 / 69.7–35.0 / 59.8 / 69.434.9 / 59.6 / 69.4

Table R2 — MCQ accuracy (%). C1 and C2 use t = 0.92.

DatasetModelA. Affirmative onlyB. eₐ − λ·eₙλC1. Single embedding −d̂C2. Mahalanobis −D
COCOCLIP52.768.0265.566.7
SigLIP-257.875.3371.371.2
AIMV253.765.01.2563.263.3
VOC2007CLIP61.186.61.579.978.3
SigLIP-268.591.61.2588.788.4
AIMV259.880.2274.173.2
MSR-VTTCLIP52.669.81.2557.057.7
SigLIP-251.573.91.2560.360.4
AIMV252.069.71.2558.660.4
Mean56.675.6–68.868.8
Correct answer's template (mean over 9 cells)A. Affirmative onlyB. eₐ − λ·eₙC1. Single embedding −d̂C2. Mahalanobis −D
Affirmation76.369.378.880.2
Negation39.577.466.464.0
Hybrid58.777.162.764.3
All56.675.668.868.8

(Continued in Part 4/6.)

Reviewer_1SWEOpenReview ↗
Rating
2(−1)
Confidence
3(+1)
Informativeness
2(−1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper is focusing on a problem that how do we represent "not" in the latent space. The authors give a sound motivation that the area of "not" shouldn't be represented as a point, because not represents a lot of things rather than just a single small area. I think it's an important problem, though the authors aren't the first to discuss it. Then the authors propose a method to measure the Mahalanobis distance for Riemannian geometry. It is used for measuring whether a data point is close to a point feature on the geometry space without additional training. This makes the negation space measurement quite transferable. Overall I think it's a good paper, and my main concern is how to empirically use it.

Critical strengths
  • The problem this paper discusses is very important. The CLIP style measurement for "not" has issues intuitively. I think this problem deserves more discussion.
  • The performance of the subspace modeling is good. SpaceVLM shows it's improvements comparing with the base model.
  • This method is quite transferrable to different VLMs, which is quite general. Although the alignment style VLMs are a bit old.
Critical weaknesses
  • Though the paper is the first to propose how to leverage the negation space of pretrained VLMs without additional training, the novelty is limited as previous works has already discussed the point feature problem, and the negation space for "not".
  • The CLIP/SigLIP style VLMs are a bit old. I'd like to know how the authors would transfer it to a modern VLM, say QwenVL. Maybe not just directly transfer the method, but how to model the negation space on those modern designs in a training-free style?
Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Rebuttal for Reviewer 1SWE

We sincerely thank Reviewer 1SWE for their positive feedback and thoughtful comments. Below, we address their concerns one by one.

Weakness 1

We agree that the limits of point embeddings have been discussed before, and we will add a paragraph on these works to the related work section. Below, we summarize the closest lines of work and how ours differs. We also add a paragraph regarding investigation of negation outside of VLMs, including in LLMs and MLLMs.

  • Region or distribution embeddings in VLMs. PCME (Chun et al., CVPR 2021) and ProbVLM (Upadhyay et al., ICCV 2023) represent an image or a caption as a probability distribution instead of a point. MERU (Desai et al., ICML 2023) and HyCoCLIP (Pal et al., ICLR 2025) represent a text as an entailment cone in hyperbolic space. These works target ambiguity (one-to-many matching) or hierarchy, not logical operators such as negation, and all of them need training (a new model, or an additional adapter in ProbVLM).
  • Negation as a region outside VLMs. Widdows (ACL 2003) models negation in text retrieval as a projection onto an orthogonal subspace, and BetaE (Ren & Leskovec, NeurIPS 2020) models negation in knowledge-graph queries with Beta distributions, because points and boxes cannot represent it. These works address text-only or knowledge-graph embeddings. The first uses flat linear subspaces, not the spherical geometry of VLMs, and the second learns the embeddings and the logical operators from scratch. Negation is also an open problem for generative MLLMs. NegVQA (Zhang et al., ACL 2025) tests 20 VLMs from seven model families, including LLaVA and Qwen2-VL, and reports a substantial accuracy drop on negated questions compared with the original questions.

To the best of our knowledge, our work is the first to bring the region view to frozen joint-embedding VLMs for a logical operator without any training (assuming using a global t=0.92 which works well as shown in our response to Reviewer gxYg). We define the region directly on the unit sphere (spherical caps), derive its center and an anisotropic distance in closed form. We also discuss how the same construction extends to other logical operators in Appendix B.

Weakness 2

As discussed earlier, NegVQA provides a benchmark that highlights the shortcomings of MLLMs in handling negation; however, it does not propose a training-free or geometric model of negation for generative MLLMs. A direct transfer of our method is not possible, because generative MLLMs do not have a single CLS embedding, as joint-embedding VLMs do. To this end, modeling the negation space inside generative MLLMs requires a deeper investigation of their internal representations, which is outside the scope of this paper. We will add this discussion and this future direction to the final version.

Could you please share some insights about how to model the negation space for MLLM in this discussion? I'm quite interested in this problem.

Reviewer_gxYgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that contrary to the prior methods, a negated caption cant be represented by a single point in joint vision language embedding space and should be rather modeled as a subspace. The method works in 3 steps starting from using an LLM to split the text into affirmation and negation, then placing a spherical cap around each part's text embedding and finally the images are scored by an anisotropic geodesic Mahalanobis distance. The method doesnt require training and the authors report gains over each of the baselines.

Critical strengths
  1. The method shows large and consistent gains without any retraining. It also generalizes well to a medical model. A drop-in fix of this scale at inference time is valuable and practical.
  2. The authors present a clean closed-form derivation for the region center, spreads and the tangent-space distance making it easy to reproduce and analyse.
Critical weaknesses
  1. The region picture doesnt match where image embeddings actually lie and the method in practice scores against a single point defeating the whole purpose
  • According the fig.3b image-caption cosines are about 0.1-0.4 which is 66-84 degs. The caps use t:0.9-0.95, a radius of only 18-26 deg. Even same-class image-image similarities fall well outside that radius so esentially no image lies inside cap(A) let alone in the intersecting region. Therefore the region membership plays no role. For every image the score is an anisotropic angular distance to one direction and that direction is the weighted combination of e_a and e_n rotated away from latter.
  • To rank images a single vector only needs to score non-N images over N image. It doesnt need to be similar to every non-N concept and a vector proportional to e_a - lambda*e_n can do that easily so statement 1 doesnt hold for ranking
  • The paper's own T2I exp shows that while using only the center point representation it still gains 32 points. This suggests that the benefit comes from a well chosen point than modeling a region
  1. Insufficient ablations to justify the source of gains. Only the method in the paper receives the LLM split of the caption; the baselines never do. Moreover the following three ablations are crucial and should have been included
  • scoring with only the affirmative part
  • linear contrast cos(e_I, e_a) − lambda*·cos(e_I, e_n)
  • cosine to d vs mahalanobis distance table 4 already shows that scoring with only affirmation might explain much of the gain since with the manual splitting, negated recall almost equals affirmative recall
  1. Running a 7B LLM on every caption for splitting is highly inefficient which also makes the comparison to the baselines a bit unfair.
Questions for authors
  1. What fraction of NegBench images actually fall inside cap(e_a)?
  2. Please report the three ablations above on retrieval and MCQ, using the same splitter: affirmative-only scoring, a linear contrast with tuned λ, and plain cosine to d. Very Crucial
  3. In retrieval, are affirmative queries also passed through the LLM? How often does the LLM hallucinates a negation?
  4. How was t chosen for each dataset, and which validation splits were used and how do results change with a single global t?
  5. Appendix B describes an extension to multiple negation types and logical operators but there is no evaluation to back it
5 comments
AuthorsOpenReview ↗

Addressing Weaknesses and Questions for Reviewer gxYg (Part 1/4)

(Part 1/4)

We sincerely thank Reviewer gxYg for their detailed and thoughtful review and for the time they devoted to it. Below, we address their concerns one by one.

Weaknesses

Weakness 1

(a) The reviewer's concerns about the distances between image and text embeddings (the modality gap) and about image–image similarity are valid. However, our approach does not contradict the modality gap, because we model only the text representation as a subspace: the problem with negation is a shortcoming of the text encoder, not the image encoder. The image only plays a role when we score the similarity between a text (a subspace) and an image (a point). Images closer to the text subspace receive higher scores, and more distant ones receive lower scores; image points do not need to lie inside the text subspace. All equations that define the membership rule (Eq. 1 to Eq. 3) are applied to text embeddings only, and the image embedding (eᵢ) is used only to measure the distance to this subspace in Eq. 4. We acknowledge that Fig. 3b may be misleading, as it suggests that images need to lie inside the cap; its purpose was only to motivate subspace modeling. We will clarify this point in the final version. Also, as Eq. 4 shows, the final Mahalanobis distance (the distance between an image point and a text subspace) is not only a weighted combination of eₐ and eₙ.

(b) With Statement 1, we aimed to motivate our idea in the general case, where we want the final image to include multiple concepts while also excluding a concept. In this case, a single embedding cannot be close to all N affirmative concepts while also excluding the negated one (particularly when the negated concept is close to the affirmative concepts). This motivated our idea in the general case, but we agree with the reviewer that for specific applications such as ranking, a single point can be sufficient, so a subspace is not strictly necessary. We will emphasize the general case we had in mind in the final version of the paper.

(c) We address this point under Weakness 2.

Weakness 2

We have run the three ablations the reviewer requested. By "scoring with only the affirmative part," we assume the reviewer means that we feed only the affirmative part to the VLM and check whether it can still identify the correct caption (MCQ) or image (retrieval). If the reviewer meant something different, please let us know.

To avoid confusion with the tables in the paper, we number the tables in this response R1–R4.

Table R1 — Negated retrieval, R@1 / R@5 / R@10 (Neg). C1 and C2 use t = 0.92.

DatasetModelAffirmative queryA. Affirmative onlyB. eₐ − λ·eₙλC1. Single embedding −d̂C2. Mahalanobis −D
COCOCLIP29.8 / 54.8 / 66.029.7 / 54.8 / 66.230.9 / 56.2 / 67.50.2529.8 / 54.9 / 66.229.8 / 54.9 / 66.2
SigLIP-248.1 / 72.6 / 80.947.8 / 72.8 / 81.148.3 / 73.4 / 81.50.2547.9 / 72.9 / 81.347.8 / 72.8 / 81.2
AIMV241.5 / 66.8 / 76.541.4 / 67.0 / 76.541.7 / 67.3 / 76.80.141.5 / 67.1 / 76.641.5 / 67.1 / 76.6
MSR-VTTCLIP28.5 / 49.6 / 61.128.9 / 52.2 / 62.428.8 / 52.4 / 62.60.0528.8 / 52.1 / 62.828.6 / 52.1 / 62.8
SigLIP-235.2 / 59.0 / 70.930.6 / 54.3 / 63.431.0 / 54.7 / 63.70.0530.3 / 54.8 / 63.630.4 / 54.2 / 64.0
AIMV231.3 / 55.9 / 65.531.3 / 56.5 / 66.631.3 / 56.7 / 66.20.131.6 / 56.7 / 65.831.5 / 56.7 / 65.8
Mean35.7 / 59.8 / 70.235.0 / 59.6 / 69.435.3 / 60.1 / 69.7–35.0 / 59.8 / 69.434.9 / 59.6 / 69.4

Table R2 — MCQ accuracy (%). C1 and C2 use t = 0.92.

DatasetModelA. Affirmative onlyB. eₐ − λ·eₙλC1. Single embedding −d̂C2. Mahalanobis −D
COCOCLIP52.768.0265.566.7
SigLIP-257.875.3371.371.2
AIMV253.765.01.2563.263.3
VOC2007CLIP61.186.61.579.978.3
SigLIP-268.591.61.2588.788.4
AIMV259.880.2274.173.2
MSR-VTTCLIP52.669.81.2557.057.7
SigLIP-251.573.91.2560.360.4
AIMV252.069.71.2558.660.4
Mean56.675.6–68.868.8
Correct answer's template (mean over 9 cells)A. Affirmative onlyB. eₐ − λ·eₙC1. Single embedding −d̂C2. Mahalanobis −D
Affirmation76.369.378.880.2
Negation39.577.466.464.0
Hybrid58.777.162.764.3
All56.675.668.868.8

(Continued in Part 2/4.)

AuthorsOpenReview ↗

Addressing Weaknesses and Questions for Reviewer gxYg (Part 2/4)

(Part 2/4)

Table R3 — Transfer MCQ → Retrieval, R@1 / R@5 / R@10 (Neg). λ is selected on mean MCQ accuracy and then frozen for retrieval. Δ is relative to tuning λ on the target cell. The other methods have no tuned hyperparameter, so their numbers match Table R1.

DatasetModelλ (from MCQ)B. eₐ − λ·eₙ (transferred)Δ for BA. Affirmative onlyC1. Single embedding −d̂C2. Mahalanobis −D
COCOCLIP215.1 / 32.6 / 42.5−15.8 / −23.6 / −25.029.7 / 54.8 / 66.229.8 / 54.9 / 66.229.8 / 54.9 / 66.2
SigLIP-21.2540.2 / 64.8 / 74.2−8.1 / −8.6 / −7.347.8 / 72.8 / 81.147.9 / 72.9 / 81.347.8 / 72.8 / 81.2
AIMV21.2529.9 / 55.1 / 66.4−11.8 / −12.2 / −10.541.4 / 67.0 / 76.541.5 / 67.1 / 76.641.5 / 67.1 / 76.6
MSR-VTTCLIP27.8 / 18.8 / 25.1−21.0 / −33.6 / −37.528.9 / 52.2 / 62.428.8 / 52.1 / 62.828.6 / 52.1 / 62.8
SigLIP-21.2520.6 / 38.1 / 46.4−10.4 / −16.6 / −17.330.6 / 54.3 / 63.430.3 / 54.8 / 63.630.4 / 54.2 / 64.0
AIMV21.2517.9 / 35.8 / 44.9−13.4 / −20.9 / −21.331.3 / 56.5 / 66.631.6 / 56.7 / 65.831.5 / 56.7 / 65.8
Mean–21.9 / 40.9 / 49.9−13.4 / −19.2 / −19.835.0 / 59.6 / 69.435.0 / 59.8 / 69.434.9 / 59.6 / 69.4

Findings:

  • Affirmative only. On retrieval, scoring with only the affirmative part is on par with our methods, because this task is a coarse-grained assessment, as described in Section 4.1 (Tasks) of the paper. MCQ, the fine-grained task, shows the difference clearly: affirmative-only reaches 56.6% on average, compared with 68.8% for our methods, and only 39.5% on negation-template captions, compared with 64.0% for D. Interestingly, it still outperforms vanilla CLIP: since the affirmative and negated concepts are no longer mixed in one embedding, the model performs well on pure affirmation captions (i.e., "<A>") and partially on hybrid captions (i.e., "<A> but not <N>"), but it performs as poorly as CLIP on pure negation captions (i.e., "not <N>").

  • Linear contrast (eₐ − λ·eₙ). This method performs well on a task when λ is tuned on that task, and in that setting it can outperform both of our methods (d̂ and D). However, it does not generalize across tasks (Table R3): the best λ ranges from 0.05 on retrieval to 3 on MCQ, a 60× difference, while our method uses a single t = 0.92 everywhere. Moreover, the linear contrast is a relaxation of d̂, with one key difference: we derive a closed-form, per-caption λ(θ) from the geometry of the model's embedding space, instead of tuning λ for each task and dataset. As the tables show, our framework generalizes with a single global threshold (t = 0.92) across all datasets, tasks (at least on NegBench), and models. We can therefore also present the closed-form derivation of λ(θ) as a side contribution of our paper, as it removes the need to tune λ for each task and dataset.

  • d̂ vs. D. We would like to clarify an important point: both d̂ (which we used for the T2I experiments) and D are main contributions of our paper. Both are derived by treating negation as a subspace in the text embedding space. The difference is that d̂ assumes an isotropic cap around the center, whereas D assumes an anisotropic one, which models the geometry of the embedding space more accurately. d̂ can therefore be seen as an approximation of D that ignores the anisotropic shape of the final subspace and treats it as an isotropic cap centered at d̂. Both work in practice, and we will clarify this alongside these ablations. In the ablation below, we explore in which situations D works better than d̂, and vice versa.

(Continued in Part 3/4.)

AuthorsOpenReview ↗

Addressing Weaknesses and Questions for Reviewer gxYg (Part 3/4)

(Part 3/4)

Since both methods reach the same overall accuracy, we analyze in more depth the samples that D predicts correctly and d̂ predicts incorrectly, and vice versa:

Table R4 — MCQ samples where d̂ and D disagree.

Correct answer's template# SamplesAcc d̂Acc −D−D-only / d̂-only correctd̂'s losses: gap (rank 2)−D's losses: gap (rank 2)P(d̂ gap > −D gap)
Affirmation914776.377.6122 / 00.16° (99%)——
Hybrid1365664.866.6312 / 740.27° (100%)0.13° (96%)0.65
Negation1303276.573.363 / 4820.06° (98%)0.21° (99%)0.33
All3583572.071.8497 / 5560.21° (100%)0.19° (98%)0.51

About the table:

  • The analysis covers the samples where d̂ and D choose different captions in the MCQ task (regardless of which one is correct), across 3 datasets (COCO, VOC2007, MSR-VTT) and 3 models (CLIP, SigLIP-2, AIMV2), i.e., 9 combinations in total, with results averaged over them.

  • We group the samples by the template of the correct answer (affirmation, negation, hybrid), as shown in the "Correct answer's template" column. "# Samples" is the total number of samples with that template across the 9 setups. "Acc d̂" and "Acc −D" are the accuracies of the two methods, as in Table R2. "−D-only / d̂-only correct" gives the number of samples where D is correct and d̂ is wrong, versus the number where d̂ is correct and D is wrong; this shows which template each method handles better. The two "gap (rank 2)" columns give the angular distance between the wrong answer chosen by each method and the correct answer. A small angle means the wrong answer is a near miss, while a large angle points to a blind spot of that method. In parentheses, we report how often the correct answer is ranked 2nd among the 4 captions (the wrong answer is, by definition, ranked 1st). The last column is the probability that the gap of d̂'s wrong answers is larger than that of D's, estimated with the Mann–Whitney U test. Note that the accuracies in Table R4 are weighted by the number of samples, whereas Table R2 averages over the 9 cells, so the two tables report slightly different numbers.

  • For captions whose correct answer is a pure affirmation, D is never worse than d̂. This is consistent with Eq. 4 in the paper, since both d̂ and D reduce to vanilla CLIP inference for pure affirmation captions. However, D penalizes the competing negated captions more strongly, which leads to better accuracy on samples whose correct answer is a pure affirmation. For hybrid captions, D also performs better, and when D is wrong, its angular gap to the correct answer is about half that of d̂ (0.13° vs. 0.27°), which suggests that D's mistakes are also closer misses. For pure negation, however, D performs worse than d̂, which can also be verified from Eq. 4. For pure negation captions, the affirmative part is the placeholder "A photo of something", whose embedding turns out to be very close to that of the negated part "a photo of <N>". This makes θ very small, so D penalizes it much more than d̂ does. Since D is clearly more reliable for hybrid captions, we expect D to perform better whenever a real affirmative object is specified in the caption (turning it into a hybrid caption). This suggests a simple rule for practical applications: if the caption is affirmative or hybrid, use D; if it is a pure negation and a placeholder must be used, use d̂.

This concludes our ablation studies. Both d̂ and D perform well on overall NegBench accuracy, but their reliability differs across caption types, so a hybrid approach that uses both could be the best choice in practice. Nevertheless, both approaches are contributions of our paper, and neither could have been derived in closed form without the subspace view: d̂ uses an isotropic subspace as an approximation, while D uses an anisotropic subspace that better models the geometry of the embedding space.

(Continued in Part 4/4.)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.