- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies the negation understanding problem in Vision-Language Models (VLMs), e.g., retrieving a picture of a dog not on grass. Specifically, it proposes a test-time steering method that uses Mahalanobis distance in the subspace to produce a refined score. However, during inference, it always requires a Large Language Model (LLM) to parse the caption, which is impractical in high-throughput VLM deployments. In addition, this paper lacks ablation studies demonstrating that the Mahalanobis distance outperforms other distance metrics or scoring functions, failing to verify its core contribution.
Critical strengths
- This paper studies a very practical problem in VLMs, i.e., negation understanding, which could benefit a wide range of real-world applications.
- The paper is well written and easy to follow.
Critical weaknesses
- The main limitation is the practicality of the proposed method. During inference, it always requires an LLM (e.g., Mistral-7B) to parse the caption, and the resulting latency and computational cost undoubtedly limit high-throughput VLM deployments. For example, in image-to-text retrieval, a VLM may need to efficiently score thousands of candidate captions.
- The paper lacks a comparison with test-time adaptation methods [1] and post-hoc correction methods [2]. In addition, the related work section would benefit from a broader discussion of negation understanding beyond embedding VLMs, e.g., including applications in MLLMs and generation models.
- One core contribution of this work is to compute the matching score using the Mahalanobis distance. However, the paper lacks ablation studies demonstrating the superiority and necessity of this metric.
- The T2I generation results suggest that \hat{d} can serve as an effective point representation for a negated caption. This appears to be in tension with the claim that point embeddings in a joint embedding space can be limiting.
[1] Negation-aware test-time adaptation for vision-language models. TPAMI 2026.
[2] What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features. ECCV 2026.
Questions for authors
- The paper states that the results of Aggarwal et al. (2026) could not be reproduced. Please clarify whether this was based on an independent reimplementation or on running the official code [1] released by the authors.
[1] Implementation of the Paper "Seeing What's Not There: Negation Understanding Needs More Than Training". ICLR 2026. URL https://github.com/ananta-nirvana-ml/negation-understanding
7 comments
Addressing Weaknesses and Questions for Reviewer hHVy (Part 2/6)
(Part 2/6)
PeakPatch [2] (Lu et al., ECCV 2026) also keeps the VLM frozen, but it differs from SpaceVLM in what it assumes and in what it needs. PeakPatch locates the loss of negation inside the text encoder: negation is separable in intermediate layers but collapses in the final layers. It therefore trains two modules (5.2M parameters) on about 1.06M CC12M caption pairs and 313K synthetic MCQ samples with LLM-generated negations: an Embedding Correction Network, which reads intermediate-layer tokens and adds a learned offset to the final text embedding, and a Score Correction Network, which adds a learned, bounded offset to the image–text score for pairwise tasks such as MCQ. SpaceVLM, in contrast, uses only the final normalized embeddings. It needs no training data, no added parameters, and no access to intermediate activations, and it derives its scoring rule in closed form, with a single global threshold t shared across tasks. Because affirmative captions reduce to vanilla CLIP scoring in SpaceVLM, its behavior on them is unchanged by construction, whereas PeakPatch alters every text embedding (e.g., −2.8 pp zero-shot accuracy on ImageNet). The two methods also represent negation differently. The corrected PeakPatch embedding is still a single point, and the authors note that, because negation is set-valued, their method recovers the affirmative/negated distinction but does not resolve a negated phrase to its complement; their own visualization shows negated captions spreading into a looser shared region rather than collapsing to a point. SpaceVLM models this region explicitly, as the intersection of an affirmative cap with the complement of a negated cap, and scores images by their distance to it. In exchange, PeakPatch needs no caption splitter at inference, whereas SpaceVLM relies on one; our splitter is modular, and Fig. 5 and Table 4 show that a 1B LLM or manual splitting yields similar gains. Empirically, the two approaches are complementary: with CLIP ViT-B/32, PeakPatch is stronger on COCO MCQ (74.3% vs. 66.7%) and on COCO negated retrieval, while SpaceVLM is stronger on VOC2007 MCQ (80.9% vs. 65.5%) and on text-to-image generation (combined accuracy 59.7% vs. 45.8%), without any training.
(Continued in Part 3/6.)
Addressing Weaknesses and Questions for Reviewer hHVy (Part 3/6)
(Part 3/6)
Regarding the broader related work, and in line with our response to Reviewer 1SWE, we will add the following discussion to the related work section:
Region or distribution embeddings in VLMs. PCME (Chun et al., CVPR 2021) and ProbVLM (Upadhyay et al., ICCV 2023) represent an image or a caption as a probability distribution instead of a point. MERU (Desai et al., ICML 2023) and HyCoCLIP (Pal et al., ICLR 2025) represent a text as an entailment cone in hyperbolic space. These works target ambiguity (one-to-many matching) or hierarchy, not logical operators such as negation, and all of them need training (a new model, or an additional adapter in ProbVLM).
Negation as a region outside VLMs. Widdows (ACL 2003) models negation in text retrieval as a projection onto an orthogonal subspace, and BetaE (Ren & Leskovec, NeurIPS 2020) models negation in knowledge-graph queries with Beta distributions, because points and boxes cannot represent it. These works address text-only or knowledge-graph embeddings. The first uses flat linear subspaces, not the spherical geometry of VLMs, and the second learns the embeddings and the logical operators from scratch. Negation is also an open problem for generative MLLMs. NegVQA (Zhang et al., ACL 2025) tests 20 VLMs from seven model families, including LLaVA and Qwen2-VL, and reports a substantial accuracy drop on negated questions compared with the original questions.
Weakness 3
As in our response to Reviewer gxYg, we provide the full ablation study here.
To avoid confusion with the tables in the paper, we number the tables in this response R1–R4.
Table R1 — Negated retrieval, R@1 / R@5 / R@10 (Neg). C1 and C2 use t = 0.92.
| Dataset | Model | Affirmative query | A. Affirmative only | B. eₐ − λ·eₙ | λ | C1. Single embedding −d̂ | C2. Mahalanobis −D |
|---|---|---|---|---|---|---|---|
| COCO | CLIP | 29.8 / 54.8 / 66.0 | 29.7 / 54.8 / 66.2 | 30.9 / 56.2 / 67.5 | 0.25 | 29.8 / 54.9 / 66.2 | 29.8 / 54.9 / 66.2 |
| SigLIP-2 | 48.1 / 72.6 / 80.9 | 47.8 / 72.8 / 81.1 | 48.3 / 73.4 / 81.5 | 0.25 | 47.9 / 72.9 / 81.3 | 47.8 / 72.8 / 81.2 | |
| AIMV2 | 41.5 / 66.8 / 76.5 | 41.4 / 67.0 / 76.5 | 41.7 / 67.3 / 76.8 | 0.1 | 41.5 / 67.1 / 76.6 | 41.5 / 67.1 / 76.6 | |
| MSR-VTT | CLIP | 28.5 / 49.6 / 61.1 | 28.9 / 52.2 / 62.4 | 28.8 / 52.4 / 62.6 | 0.05 | 28.8 / 52.1 / 62.8 | 28.6 / 52.1 / 62.8 |
| SigLIP-2 | 35.2 / 59.0 / 70.9 | 30.6 / 54.3 / 63.4 | 31.0 / 54.7 / 63.7 | 0.05 | 30.3 / 54.8 / 63.6 | 30.4 / 54.2 / 64.0 | |
| AIMV2 | 31.3 / 55.9 / 65.5 | 31.3 / 56.5 / 66.6 | 31.3 / 56.7 / 66.2 | 0.1 | 31.6 / 56.7 / 65.8 | 31.5 / 56.7 / 65.8 | |
| Mean | 35.7 / 59.8 / 70.2 | 35.0 / 59.6 / 69.4 | 35.3 / 60.1 / 69.7 | – | 35.0 / 59.8 / 69.4 | 34.9 / 59.6 / 69.4 |
Table R2 — MCQ accuracy (%). C1 and C2 use t = 0.92.
| Dataset | Model | A. Affirmative only | B. eₐ − λ·eₙ | λ | C1. Single embedding −d̂ | C2. Mahalanobis −D |
|---|---|---|---|---|---|---|
| COCO | CLIP | 52.7 | 68.0 | 2 | 65.5 | 66.7 |
| SigLIP-2 | 57.8 | 75.3 | 3 | 71.3 | 71.2 | |
| AIMV2 | 53.7 | 65.0 | 1.25 | 63.2 | 63.3 | |
| VOC2007 | CLIP | 61.1 | 86.6 | 1.5 | 79.9 | 78.3 |
| SigLIP-2 | 68.5 | 91.6 | 1.25 | 88.7 | 88.4 | |
| AIMV2 | 59.8 | 80.2 | 2 | 74.1 | 73.2 | |
| MSR-VTT | CLIP | 52.6 | 69.8 | 1.25 | 57.0 | 57.7 |
| SigLIP-2 | 51.5 | 73.9 | 1.25 | 60.3 | 60.4 | |
| AIMV2 | 52.0 | 69.7 | 1.25 | 58.6 | 60.4 | |
| Mean | 56.6 | 75.6 | – | 68.8 | 68.8 |
| Correct answer's template (mean over 9 cells) | A. Affirmative only | B. eₐ − λ·eₙ | C1. Single embedding −d̂ | C2. Mahalanobis −D |
|---|---|---|---|---|
| Affirmation | 76.3 | 69.3 | 78.8 | 80.2 |
| Negation | 39.5 | 77.4 | 66.4 | 64.0 |
| Hybrid | 58.7 | 77.1 | 62.7 | 64.3 |
| All | 56.6 | 75.6 | 68.8 | 68.8 |
(Continued in Part 4/6.)
Addressing Weaknesses and Questions for Reviewer hHVy (Part 1/6)
(Part 1/6)
We sincerely thank Reviewer hHVy for their detailed and thoughtful review. Below, we address their concerns one by one.
Weakness 1
As in our response to Reviewer gxYg, we emphasize that our paper's contribution is modeling negation as a subspace in the embedding space, once its affirmative and negated parts are available. For evaluation, we need to handle natural negation forms in NegBench, such as implicit negation, and we use an LLM for this, since rule-based methods cannot easily identify them. However, any other method can be used instead, including a rule-based one for simpler negated captions, to improve speed in practice: the splitter is modular and can be replaced by any other approach. So, we do not always need an LLM at inference time. In this paper, we used LLMs to keep the evaluation simple, since our contribution lies elsewhere, but practical applications can use a smaller LLM, a rule-based method, or a hybrid of the two for better throughput and efficiency. Moreover, as a side analysis of caption splitting, Fig. 5 shows that a 7B model is not necessary: 1B models give the same performance with much lower latency, and a 360M LLM (of the same order of magnitude as CLIP itself) gives slightly lower performance but is still a much better choice than vanilla CLIP.
Weakness 2
We thank the reviewer for pointing us to these recent post-hoc works on negation. Both are concurrent with our submission: [1] was published in TPAMI in October 2026, after we submitted our work to ICLR 2027, and, since we compare only with peer-reviewed methods, [2] became available for comparison only when the ECCV 2026 proceedings were released in September 2026, close to the ICLR deadline. It was therefore not feasible for us to compare with them in the submitted version. Below, we discuss both works and how their methodology differs from ours.
NEAT [1] (Han et al., TPAMI 2026) also targets negation in CLIP-like models without large-scale post-training, but its methodology differs from ours in three main ways. First, NEAT is a test-time adaptation method. It attributes the failure to a dual-concept distribution shift between affirmative and negated captions, and it corrects this shift by updating the LayerNorm parameters of the text encoder with gradient descent on batches of unlabeled test data. SpaceVLM, in contrast, updates no parameters and computes no gradients (with a single global t, which we show to work well in our response to Reviewer gxYg). It scores each query independently, so it needs no test-time batch, optimizer, or learning rate. Second, after adaptation, NEAT still represents each caption as a single point embedding, and the shift it corrects is defined over distributions of such points. SpaceVLM instead represents a negated caption as a region on the unit sphere, namely the intersection of an affirmative cap with the complement of a negated cap, and scores images by a closed-form anisotropic geodesic distance to this region. Third, like SpaceVLM, NEAT uses an LLM to split a negated caption into its affirmative and negated parts, but the two methods use these parts differently. NEAT combines them with a heuristic product rule, (t^P·v)(1 − α[t^N·v]₊), with a fixed trade-off α, and uses this rule only to select pseudo-targets for its losses; it also needs an LLM-generated reversed caption. In addition, because SpaceVLM leaves both encoders unchanged, its behavior on affirmative captions is unchanged by construction, whereas NEAT reports changes of −4.75% to +4.91% in standard retrieval. Finally, on COCO MCQ with CLIP ViT-B/32, NEAT improves accuracy from 39.25% to 48.41%, while SpaceVLM improves it from 39.2% to 66.7% (Table 2).
(Continued in Part 2/6.)