- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To improve the spatial perception of CLIP, this paper proposes adding extra spatial caption to training images. However, only taking the spatial caption as positive sample undermines the zero-shot performance. To address this issue, this paper rather than take the sampled point on the arc between the template caption and spatial caption. In this way, both the zero-shot performance and spatial perception are improved. The experiments demonstrate the effectiveness of the proposed method.
Critical strengths
This paper proposes a novel and cheap approach to improve the spatial perception of CLIP. It does not require extra teacher model during training nor extra modules.
Critical weaknesses
The ablation study are not sufficient. This paper only validate the effectiveness of spatial caption while not exclude other factors. For example, this paper should have the same MLLM generate captions for the image that do not contain spatial information, or have the MLLM generate captions but not deliberately emphasize spatial information, and then compare them with captions that deliberately emphasize spatial information. This can rule out whether the gain is due to more caption information. This experiment is crucial to the argument of this paper.
In addition, the method in this paper relies on the annotation of an external stronger MLLM, which is similar to distillation. I cannot confirm whether this approach is meaningful because its performance improvement depends on a stronger MLLM, and its purpose is to generate a stronger MLLM, unless this paper can demonstrate that the performance ceiling can be broken through continuous self-iteration.
Questions for authors
- Why not perform ablation experiments that optimize two positive samples simultaneously? Alternating updates are not equivalent to optimizing two positive samples simultaneously.
- How much gain is retained if the text encoder is frozen and only the image encoder is fine tuned?
3 comments
part 2
Q2 — how much gain is retained with a frozen text encoder? Done. The counting gain is retained in full; everything else needs the text encoder.
| Text encoder | Text target | Spatial mean | CLEVR-Counts | CLEVR-Dist. | ZS-11 | Flickr T→I |
|---|---|---|---|---|---|---|
| Trained | Template-only | 32.06 ± 1.01 | 16.66 ± 1.28 | 24.55 ± 0.03 | 69.04 ± 0.20 | 66.08 ± 0.24 |
| Trained | Arc | 41.22 ± 1.30 | 26.86 ± 1.86 | 25.99 ± 0.42 | 69.62 ± 0.22 | 67.12 ± 0.52 |
| Frozen | Template-only | 33.10 | 14.93 | 15.10 | 67.34 | 70.46 |
| Frozen | Arc | 35.45 | 27.29 | 20.85 | 65.65 | 59.56 |
With the text encoder frozen, arc smoothing still improves the four-task spatial mean over frozen-text template-only training, by +2.35 points against +9.16 when both encoders are trained: about a quarter of the gain is retained. The split is informative. The counting gain is retained in full (CLEVR-Counts 27.29, +12.4 over the frozen-text baseline and level with the two-tower model), and CLEVR-Distance gains +5.8. What is lost is everything that depends on the text side: ZS-11 falls by 1.7 and text-to-image R@1 by 10.9 / 10.3 on Flickr30K / MSCOCO, consistent with a frozen text encoder that cannot move its caption embeddings toward the targets the image encoder is trained on.
This is the premise of the paper measured directly. Existing methods "give the text encoder no spatial signal" (Abstract); letting gradients flow through the text branch is what turns a counting gain into a gain on all four spatial tasks, on retrieval and on transfer at once.
These are the experiments the review identifies as crucial, and each one came out in favour of the method. We hope the reviewer will find them sufficient to move the rating above the acceptance threshold, and we are glad to run anything further.
A gentle reminder
Dear Reviewer XXqr,
As the discussion period is closing soon, we kindly draw your attention to our response. In it we report all three experiments you asked for: the caption-content control, simultaneous positives and the frozen text encoder. We would be grateful to know whether they address your concerns.
Sincerely,
Authors
Response to Reviewer XXqr
We thank the reviewer for recognising the approach as novel and cheap, and for a precise list of what would change the assessment. We ran all three experiments the review asks for (W1, Q1, Q2), and we answer W2 with evidence that is already in the paper.
W1 — is the gain due to spatial content, or just to more caption information? We ran exactly this ablation. It is the spatial content.
Same captioner (CapRL), same images, same arc law Beta(0.2,0.2), same recipe. Only the content of the caption changes (Table R3):
Captions that do not emphasise spatial information add +1.6 points over no captions at all; spatial captions add +9.2. More caption information therefore accounts for at most 1.6 of the 9.2 points. Removing only the relation terms from our own captions costs 5.0 points, while object names, count words and colour words are unchanged.
Neither control keeps zero-shot transfer or retrieval at the template-only level, whereas spatial captions keep or improve every column. The matched control already in the paper points the same way from the other side: balanced endpoints use the very same spatial captions at the same exposure and reach 36.92, so the information in the captions is not sufficient without targets inside the arc either (Table R2).
W2 — the method relies on a stronger MLLM, similar to distillation; is it meaningful without self-iteration? The goal is a better encoder, not a better captioner, and the encoder it produces already lifts an MLLM above its released version.
A different product. The output is a CLIP dual encoder. It serves zero-shot classification and retrieval in a single forward pass, which an MLLM does not, and it is the frozen visual front-end of MLLMs such as LLaVA-1.5, whose blind spots "cannot be trained away downstream" (Introduction). The MLLM is used once, offline, to write captions; no second network runs during training or inference. The existing alternatives (DIVA, GenHancer, KUEA) also depend on a second large model, and run it at every training step.
Not circular. Appendix C ("Is an MLLM captioner with a CLIP-like vision tower circular?") addresses this directly. MLLMs perceive substantially more than CLIP with the same vision encoder and weights (Li et al., 2025a), because they read patch tokens that CLIP's pooled similarity never surfaces. Arc smoothing moves that information into the pooled embedding and into the text encoder.
The ceiling is crossed downstream. Plugging our encoder into LLaVA-1.5 under a matched LoRA protocol gives 65.54 average VQA, against 63.33 for the released LLaVA-1.5 and 62.52 for CLIP + LoRA, with +5.0 on TallyQA counting (Table 2). That is one full turn of the loop the reviewer describes, shown across model families: captions written by a Qwen-family captioner produce an encoder that makes LLaVA-1.5 better than the same model on its original encoder. We do not claim repeated self-iteration; it is the natural next step and we will say so in the conclusion.
Not tied to one strong captioner, and not explained by captioner strength. Qwen2.5-VL-7B captions give the same profile (42.21 spatial zero-shot, 69.55 ZS-11; Table 1). And the same CapRL captions give 36.92 when used as endpoints and 12.5 points less than arc smoothing when used as simultaneous positives (Tables R2–R3): what matters is what the captions say and how they enter the loss.
Q1 — optimising two positives simultaneously instead of alternating. Done. It is below arc smoothing.
We agree that alternating and simultaneous optimisation are different, and we added a dual-objective run in which the template and the spatial caption are both positives in every step, with no interpolation (Table R3). Relative to it, arc smoothing is +12.5 points on the four-task spatial mean (+7.0 on CLEVR-Counts), +3.5 on ZS-11, and +10.5 / +11.2 text-to-image R@1 on Flickr30K / MSCOCO, at a cost of 0.7 ImageNet points. The dual objective is also 8.2 spatial points below the alternating baseline (balanced endpoints). Simultaneous positives fit both captions but never supervise a target between them. That missing piece is the interior term isolated by Proposition 2, and it is where the gain comes from.
(response continued in the next comment)