acceptodds
Under review as a conference paper at ICLR 2027

Arc-Smoothed Caption Supervision for Fine-Grained Spatial CLIP

Abstract

CLIP is the image encoder behind zero-shot recognition and many multimodal large language models (MLLMs), yet it is weak at fine-grained spatial perception such as counting objects and judging their positions and relations. Therefore, classifiers and MLLMs that reuse its encoders inherit this weakness. Existing remedies fine-tune the image encoder against a second large model placed in the training loop, such as a frozen diffusion model or a vision-only teacher, and give the text encoder no spatial signal. We propose arc-smoothed caption supervision, a simple alternative whose training targets are all points in CLIP's own image–text embedding space and which uses no other network during training or inference. For each training image, we generate a spatial caption once, offline. During fine-tuning, we move along the great-circle arc between the embeddings of the class template and the spatial caption to a position drawn from a U-shaped Beta law, and use the resulting point as the positive target of CLIP's standard contrastive loss. Targets inside the arc are essential: using the same two captions as separate positives at the same exposure gives lower spatial zero-shot accuracy than template-only fine-tuning. We show that the mixing law trades fidelity to the two caption objectives against an interior term that contrastively supervises the midpoint between each image's template and its own caption, and controlled comparisons are consistent with both sides of this trade-off. On CLIP ViT-L/14, relative to matched template-only fine-tuning and across two captioners, arc smoothing improves spatial zero-shot accuracy by – points and text-to-image retrieval R@1 by – points, while keeping zero-shot transfer on datasets at the template-only level. Used as the visual encoder of LLaVA-1.5, it improves average VQA accuracy by points, including points on TallyQA counting, suggesting that spatial information supplied through language during encoder fine-tuning carries over to downstream multimodal reasoning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
multi-modal language models, CLIP, contrastive learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_XXqrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To improve the spatial perception of CLIP, this paper proposes adding extra spatial caption to training images. However, only taking the spatial caption as positive sample undermines the zero-shot performance. To address this issue, this paper rather than take the sampled point on the arc between the template caption and spatial caption. In this way, both the zero-shot performance and spatial perception are improved. The experiments demonstrate the effectiveness of the proposed method.

Critical strengths

This paper proposes a novel and cheap approach to improve the spatial perception of CLIP. It does not require extra teacher model during training nor extra modules.

Critical weaknesses

The ablation study are not sufficient. This paper only validate the effectiveness of spatial caption while not exclude other factors. For example, this paper should have the same MLLM generate captions for the image that do not contain spatial information, or have the MLLM generate captions but not deliberately emphasize spatial information, and then compare them with captions that deliberately emphasize spatial information. This can rule out whether the gain is due to more caption information. This experiment is crucial to the argument of this paper.

In addition, the method in this paper relies on the annotation of an external stronger MLLM, which is similar to distillation. I cannot confirm whether this approach is meaningful because its performance improvement depends on a stronger MLLM, and its purpose is to generate a stronger MLLM, unless this paper can demonstrate that the performance ceiling can be broken through continuous self-iteration.

Questions for authors
  1. Why not perform ablation experiments that optimize two positive samples simultaneously? Alternating updates are not equivalent to optimizing two positive samples simultaneously.
  2. How much gain is retained if the text encoder is frozen and only the image encoder is fine tuned?
3 comments
AuthorsOpenReview ↗

Response to Reviewer XXqr

We thank the reviewer for recognising the approach as novel and cheap, and for a precise list of what would change the assessment. We ran all three experiments the review asks for (W1, Q1, Q2), and we answer W2 with evidence that is already in the paper.

W1 — is the gain due to spatial content, or just to more caption information? We ran exactly this ablation. It is the spatial content.

Same captioner (CapRL), same images, same arc law Beta(0.2,0.2), same recipe. Only the content of the caption changes (Table R3):

CapRL captions, same arc lawSpatial meanCLEVR-CountsZS-11Flickr T→ICOCO T→I
No captions (template-only)32.06 ± 1.0116.66 ± 1.2869.04 ± 0.2066.08 ± 0.2439.32 ± 0.22
Neutral prompt, no spatial emphasis33.66 ± 0.6020.31 ± 1.3866.35 ± 0.1461.40 ± 0.4034.63 ± 0.10
Spatial prompt, relation terms removed36.2720.8666.5363.0834.27
Spatial prompt (ours)41.22 ± 1.3026.86 ± 1.8669.62 ± 0.2267.12 ± 0.5240.68 ± 0.05

Captions that do not emphasise spatial information add +1.6 points over no captions at all; spatial captions add +9.2. More caption information therefore accounts for at most 1.6 of the 9.2 points. Removing only the relation terms from our own captions costs 5.0 points, while object names, count words and colour words are unchanged.

Neither control keeps zero-shot transfer or retrieval at the template-only level, whereas spatial captions keep or improve every column. The matched control already in the paper points the same way from the other side: balanced endpoints use the very same spatial captions at the same exposure and reach 36.92, so the information in the captions is not sufficient without targets inside the arc either (Table R2).

W2 — the method relies on a stronger MLLM, similar to distillation; is it meaningful without self-iteration? The goal is a better encoder, not a better captioner, and the encoder it produces already lifts an MLLM above its released version.

A different product. The output is a CLIP dual encoder. It serves zero-shot classification and retrieval in a single forward pass, which an MLLM does not, and it is the frozen visual front-end of MLLMs such as LLaVA-1.5, whose blind spots "cannot be trained away downstream" (Introduction). The MLLM is used once, offline, to write captions; no second network runs during training or inference. The existing alternatives (DIVA, GenHancer, KUEA) also depend on a second large model, and run it at every training step.

Not circular. Appendix C ("Is an MLLM captioner with a CLIP-like vision tower circular?") addresses this directly. MLLMs perceive substantially more than CLIP with the same vision encoder and weights (Li et al., 2025a), because they read patch tokens that CLIP's pooled similarity never surfaces. Arc smoothing moves that information into the pooled embedding and into the text encoder.

The ceiling is crossed downstream. Plugging our encoder into LLaVA-1.5 under a matched LoRA protocol gives 65.54 average VQA, against 63.33 for the released LLaVA-1.5 and 62.52 for CLIP + LoRA, with +5.0 on TallyQA counting (Table 2). That is one full turn of the loop the reviewer describes, shown across model families: captions written by a Qwen-family captioner produce an encoder that makes LLaVA-1.5 better than the same model on its original encoder. We do not claim repeated self-iteration; it is the natural next step and we will say so in the conclusion.

Not tied to one strong captioner, and not explained by captioner strength. Qwen2.5-VL-7B captions give the same profile (42.21 spatial zero-shot, 69.55 ZS-11; Table 1). And the same CapRL captions give 36.92 when used as endpoints and 12.5 points less than arc smoothing when used as simultaneous positives (Tables R2–R3): what matters is what the captions say and how they enter the loss.

Q1 — optimising two positives simultaneously instead of alternating. Done. It is below arc smoothing.

We agree that alternating and simultaneous optimisation are different, and we added a dual-objective run in which the template and the spatial caption are both positives in every step, with no interpolation (Table R3). Relative to it, arc smoothing is +12.5 points on the four-task spatial mean (+7.0 on CLEVR-Counts), +3.5 on ZS-11, and +10.5 / +11.2 text-to-image R@1 on Flickr30K / MSCOCO, at a cost of 0.7 ImageNet points. The dual objective is also 8.2 spatial points below the alternating baseline (balanced endpoints). Simultaneous positives fit both captions but never supervise a target between them. That missing piece is the interior term isolated by Proposition 2, and it is where the gain comes from.

(response continued in the next comment)

Q2 — how much gain is retained with a frozen text encoder? Done. The counting gain is retained in full; everything else needs the text encoder.

Text encoderText targetSpatial meanCLEVR-CountsCLEVR-Dist.ZS-11Flickr T→I
TrainedTemplate-only32.06 ± 1.0116.66 ± 1.2824.55 ± 0.0369.04 ± 0.2066.08 ± 0.24
TrainedArc41.22 ± 1.3026.86 ± 1.8625.99 ± 0.4269.62 ± 0.2267.12 ± 0.52
FrozenTemplate-only33.1014.9315.1067.3470.46
FrozenArc35.4527.2920.8565.6559.56

With the text encoder frozen, arc smoothing still improves the four-task spatial mean over frozen-text template-only training, by +2.35 points against +9.16 when both encoders are trained: about a quarter of the gain is retained. The split is informative. The counting gain is retained in full (CLEVR-Counts 27.29, +12.4 over the frozen-text baseline and level with the two-tower model), and CLEVR-Distance gains +5.8. What is lost is everything that depends on the text side: ZS-11 falls by 1.7 and text-to-image R@1 by 10.9 / 10.3 on Flickr30K / MSCOCO, consistent with a frozen text encoder that cannot move its caption embeddings toward the targets the image encoder is trained on.

This is the premise of the paper measured directly. Existing methods "give the text encoder no spatial signal" (Abstract); letting gradients flow through the text branch is what turns a counting gain into a gain on all four spatial tasks, on retrieval and on transfer at once.

These are the experiments the review identifies as crucial, and each one came out in favour of the method. We hope the reviewer will find them sufficient to move the rating above the acceptance threshold, and we are glad to run anything further.

AuthorsOpenReview ↗

A gentle reminder

Dear Reviewer XXqr,

As the discussion period is closing soon, we kindly draw your attention to our response. In it we report all three experiments you asked for: the caption-content control, simultaneous positives and the frozen text encoder. We would be grateful to know whether they address your concerns.

Sincerely,

Authors

Reviewer_ZWBDOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Arc-Smoothed Caption Supervision, a method for improving CLIP's fine-grained spatial understanding, particularly counting, object positioning, and spatial relationships. The authors argue that existing approaches often require additional large models during training and do not directly provide spatial supervision to CLIP's text encoder. The proposed method generates spatial captions offline and uses spherical linear interpolation (SLERP) between the embeddings of class templates and spatial captions. The interpolation coefficient is sampled from a Beta distribution, allowing the model to learn spatial information while preserving its original zero-shot classification capabilities. Both image and text encoders are fine-tuned using CLIP's standard contrastive loss. The authors further provide a theoretical analysis explaining how interpolation balances supervision from class templates and spatial captions. Experiments using CLIP ViT-L/14 show improvements of approximately 3.0–3.8 percentage points in spatial zero-shot accuracy and 1.4–2.1 points in text-to-image retrieval over template-only fine-tuning. Integrating the resulting visual encoder into LLaVA-1.5 also improves average VQA accuracy by approximately 3 points.

Critical strengths

The paper has the following strengths -

  1. It is a well written paper and is easy to follow.
  2. Problem they are solving is timely.
Critical weaknesses

The paper has the following weaknesses -

  1. Limited methodological novelty and insufficient comparisons with competitive baselines - I find the methodological novelty relatively limited, as the proposed approach primarily applies existing spherical interpolation and Beta-distributed sampling techniques to combine class-template and spatial-caption embeddings. Although Section 2 discusses relevant approaches such as LaCLIP, VeCLIP, Long-CLIP, FLAIR, and DAC, Table 1 (Section 7.1) does not provide direct comparisons against these methods under matched training settings. Instead, most controlled comparisons involve relatively simple caption-mixing and interpolation strategies. I believe stronger comparisons with existing caption-based CLIP adaptation methods are necessary to establish whether the proposed interpolation strategy offers a substantial methodological improvement.

  2. Limited experimental generalization and insufficient statistical validation - I am not convinced that the experimental evidence sufficiently establishes the generalizability of the proposed method. In Section 6, all experiments are conducted using CLIP ViT-L/14, without evaluating other backbone architectures or vision-language encoders such as SigLIP. Furthermore, Table 1 (Section 7.1) reports primarily single-run results, making it difficult to assess whether the reported improvements of 3.0–3.8 points in spatial zero-shot accuracy are statistically reliable. Although Section 7.2 studies different interpolation distributions, this analysis primarily examines ImageNet accuracy rather than the spatial benchmarks central to the paper. I would like to see multi-seed evaluations, additional architectures, and comprehensive spatial performance comparisons across different mixing coefficients to substantiate the method's robustness.

Questions for authors

See weaknesses section.

4 comments
AuthorsOpenReview ↗

Response to Reviewer ZWBD

We thank the reviewer for the positive assessment of the writing and of the problem. Both weaknesses ask for specific evidence, and we now provide it (Tables R1–R4).

W1(a) — limited novelty beyond slerp and Beta sampling. The contribution is the supervision signal and the account of why it works, not the interpolation operator.

The paper states that slerp and mixup-style interpolation are established tools (Introduction; Section 2). Three things are new.

(i) The signal. The targets lie between two descriptions of the same image, one of which is the classifier the model is evaluated with. Mixup combines different examples and their labels, and Geodesic Multi-Modal Mixup applies slerp across the modality gap to create hard negatives; neither produces this signal.

(ii) The analysis. Proposition 2 is an exact identity that splits any symmetric law into the endpoint objective plus an interior term, and Proposition 1 gives the endpoint fidelity that the law trades against it. Lemma 1 shows why slerp is required: its coefficient means the same fractional position on every arc, which the normalised chord does not provide when caption angles vary as widely as they do here (same-image cosine 0.35 ± 0.15, Section 4.1).

(iii) The evidence that the interior is the active ingredient. With captions and exposure identical, arc smoothing beats balanced endpoints by +4.3 spatial points and +3.2 on ZS-11 (Table R2), beats optimising both captions as simultaneous positives by +12.5 spatial points (Table R3), and slerp beats normalised linear interpolation on all six metrics of Table 1 (41.36 vs 36.99 spatial zero-shot, 69.96 vs 64.42 on ZS-11).

W1(b) — no matched comparison with LaCLIP, VeCLIP, Long-CLIP, FLAIR, DAC. The way each of them uses a caption is in our tables under a matched recipe, and arc smoothing beats all three.

These systems differ from ours in data scale and architecture: LaCLIP and VeCLIP act at pretraining on web-scale pairs, and Long-CLIP and FLAIR change the text input length or the pooling. Their released checkpoints therefore cannot be compared under a matched recipe. What can be matched is how each of them uses an additional caption, which is exactly the variable our method changes. Section 2 already maps caption mixing to our balanced-endpoint baseline, and Tables R2–R3 now cover all three mechanisms with the same backbone, data, captions and schedule:

How the extra caption is usedMatched-recipe analogueSpatial meanZS-11ImageNet
Caption replaces the short text (dense-caption fine-tuning, DAC-style)Spatial-caption-only35.85 ± 3.2562.61 ± 0.3277.58 ± 0.06
Caption sampled in place of the original text (LaCLIP, VeCLIP)Balanced endpoints36.92 ± 1.0066.38 ± 0.1486.23 ± 0.01
Long and short captions as simultaneous positives (Long-CLIP, FLAIR)Dual objective, no interpolation28.7666.0986.26
Target between the two captions (ours)Arc, Beta(0.2,0.2)41.22 ± 1.3069.62 ± 0.2285.53 ± 0.03

Arc smoothing is the only row that improves spatial accuracy and keeps ZS-11 at the template-only level. Against complete external systems for spatial CLIP, Table 1 and Appendix Tables 3–5 compare with KUEA (and DIVA in Table 3): +2.2 spatial zero-shot, +4.4 spatial probe and +3.4 ZS-11 over KUEA, which is label-free, with no second network in the training loop.

W2(a) — single-run results. Now four seeds; the spatial gain is about seven times the seed deviation.

4 seeds eachSpatial meanCLEVR-CountsGTSRBZS-11Flickr T→ICOCO T→I
Template-only32.06 ± 1.0116.66 ± 1.2835.63 ± 3.5269.04 ± 0.2066.08 ± 0.2439.32 ± 0.22
Arc, Beta(0.2,0.2)41.22 ± 1.3026.86 ± 1.8653.46 ± 0.5169.62 ± 0.2267.12 ± 0.5240.68 ± 0.05
Difference+9.16+10.20+17.83+0.58+1.04+1.36

The spatial gain (+9.2) is about seven times the larger of the two seed standard deviations (1.3). All four spatial tasks improve (SVHN +7.2 and CLEVR-Distance +1.4 in addition to the columns shown), retrieval improves on both datasets, and ZS-11 is preserved. The cost is 0.9 ImageNet points, as stated in the paper. A second captioner (Qwen2.5-VL) reaches the same level (42.21 spatial zero-shot, 69.55 ZS-11; Table 1), and the encoder transfers to LLaVA-1.5 (+3.0 average VQA, +5.0 TallyQA; Table 2).

(response continued in the next comment)

W2(b) — other backbones and SigLIP. Nothing in the method is specific to ViT-L/14.

We evaluate ViT-L/14 because it is the visual encoder of LLaVA-1.5 and the setting of KUEA and DIVA, which keeps Tables 1 and 2 matched; the scope is stated in Limitations. The method changes only the text target of the contrastive loss and adds no module. Lemma 1 and Proposition 1 are purely geometric, and Eq. (9) holds for any smooth contrastive loss, so the method and its analysis apply unchanged to other CLIP backbones and to SigLIP. A second-backbone run is in progress which needs much more GPU time because of baselines lacking this new backbone results. We will try to post it in this thread as soon as it completes.

W2(c) — spatial performance across mixing coefficients. Done: Beta(0.2,0.2) is the best law in the sweep on spatial accuracy and on transfer.

Mixing lawImageNetZS-11Spatial meanCLEVR-CountsCLEVR-Dist.
Balanced endpoints86.23 ± 0.0166.38 ± 0.1436.92 ± 1.0021.73 ± 0.9724.66 ± 0.21
Beta(0.1,0.1)85.9866.6137.7424.4824.79
Beta(0.2,0.2)85.53 ± 0.0369.62 ± 0.2241.22 ± 1.3026.86 ± 1.8625.99 ± 0.42
Beta(0.5,0.5)81.2466.0036.0822.2725.38
Beta(1,1), uniform75.2363.4131.9519.6923.57

The four-task spatial mean, CLEVR-Counts, CLEVR-Distance and ZS-11 all peak at Beta(0.2,0.2), while ImageNet decreases monotonically with β, as Section 5.3 predicts. The sweep is no longer an ImageNet-only analysis: the operating point of the paper is the maximum on the spatial benchmarks that motivate the method.

We hope the multi-seed results, the matched mechanism comparison and the spatial sweep address both weaknesses, and we would be grateful if the reviewer would consider raising the score.

Reviewer_ZWBDOpenReview ↗

Final justification

The authors have answered all my queries. A couple of important works related to this idea are not cited. For instance -

  1. Sarkar A. et. al., Leveraging Data to Say No: Memory Augmented Plug-and-Play Selective Prediction.
  2. Tong S. et. al., Eyes wide shut? exploring the visual shortcomings of multimodal llms.

I recommend accepting this paper, provided the authors cite the relevant works mentioned above and incorporate the rebuttal results into the camera-ready version.

Dear Reviewer ZWBD,

Thank you for your careful reading and for recommending acceptance. We will cite both works and include all rebuttal results in the camera-ready version.

Sincerely,

Authors

Reviewer_1GNvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether CLIP can acquire more fine-grained visual information without sacrificing the class-template interface used for zero-shot classification. Each ImageNet training image is paired with a class template and a spatial caption generated offline. The two text embeddings define a spherical arc. A target sampled from this arc, using Slerp and a symmetric Beta distribution, replaces the usual text target in CLIP's contrastive loss; both encoders are fine-tuned. The paper derives properties of the sampling distribution, separating supervision near the caption endpoints from terms contributed by interior targets. On ViT-L/14, the method improves the four-task spatial zero-shot average by 3.0 to 3.8 points over template-only fine-tuning, maintains similar transfer accuracy on 11 datasets, and improves text-to-image retrieval. A downstream LLaVA-1.5 experiment reports gains on VQA, particularly counting.

Critical strengths
  1. The experiments address the balance between spatial performance and transfer. The results show that spatial captions can improve some retrieval tasks while reducing transfer accuracy, and that alternating spatial captions with class templates does not fully recover the desired balance. The arc-sampling results offer a promising alternative within this comparison.

  2. The method retains the encoder architecture and inference procedure, which supports its practical appeal. The theoretical analysis also clarifies how the sampling law changes the contrastive objective. Comparisons with label-free methods such as KUEA provide useful context, while the matched supervised baselines offer the clearest evidence for the proposed design.

  3. Table 1 compares sampling strategies under a matched training recipe. Balanced endpoint sampling reaches 37.14% on the four-task spatial zero-shot average, while arc sampling with CapRL captions reaches 41.36%. Since both settings use the same captions, this comparison helps isolate the effect of the sampling strategy and provides stronger evidence for the design than comparisons with frozen CLIP or template-only fine-tuning alone.

Critical weaknesses
  1. The evaluation could more directly assess spatial relations. Figure 1 uses a horizontal flip to illustrate a key challenge: object identities remain unchanged while their spatial relation changes. The current evaluation provides useful evidence on recognition, counting, distance estimation, and downstream performance, but does not directly test whether the model distinguishes relational reversals such as left of and right of when object identity, count, and appearance are controlled. In particular, SVHN and GTSRB primarily evaluate fine-grained recognition, while CLEVR-Counts and CLEVR-Distance focus on counting and distance. The gains on LLaVA tasks, including TallyQA, offer encouraging downstream evidence, though they do not by themselves isolate relational sensitivity. A controlled evaluation of relational reversals would help clarify how well the method addresses the failure mode illustrated in Figure 1 and would better support the paper’s framing of its spatial contribution.

  2. Because Slerp and Beta sampling are established techniques, the paper’s main contribution appears to lie in its choice of endpoints and the evidence connecting them to spatial performance. The endpoint control suggests that, under the proposed recipe, a continuous target distribution can outperform either exact endpoint. However, this comparison does not isolate whether the improvement arises from learning spatial relations or from supervision near the arc midpoint. Spatial captions may also contribute image-specific object and attribute information. In addition, endpoint sampling produces batches trained exactly on class templates, whereas Beta sampling almost surely retains some caption-specific signal in every batch. This difference could affect instance discrimination or competition among same-class images in InfoNCE, independently of spatial binding.

  3. Proposition 2 provides an exact identity for the loss, but interpreting it as evidence for midpoint supervision depends on a leading-order term that the appendix notes may not dominate at CLIP’s temperature. Figure 2 compares sampling laws using ImageNet accuracy, rather than the spatial metrics that motivate the method. Further evidence connecting the sampling distribution to those spatial outcomes would help clarify the mechanism behind the reported gains.

Questions for authors
  1. Could the authors report accuracy and the score margin between correct and relation-reversed captions on image–text pairs that differ only in a spatial relation, such as left/right or above/below, for template-only, balanced-endpoint, and arc training? This controlled comparison would provide a direct assessment of spatial-relation sensitivity alongside the aggregate recognition results.

  2. It would be informative to assess how the improvement changes when spatial relation terms are removed or swapped while object names and descriptive detail are held constant. A comparison with attribute-focused captions of similar detail could also help distinguish the contribution of relational information from the broader benefits of image-specific targets.

  3. Figure 2 reports ImageNet performance across Beta sampling distributions. Could the authors also report CLEVR-Counts and CLEVR-Distance results across the same sweep? Comparing these results with a two-point distribution concentrated just inside the endpoints could help clarify whether the spatial effects are associated with sampling across the interior of the arc or also arise from small shifts away from the endpoint targets.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 1GNv

We thank the reviewer for the careful reading and for identifying the matched endpoint comparison as the paper's strongest evidence. That comparison holds across seeds: with the same captions at the same exposure, balanced endpoints reach 36.92 ± 1.00 and arc smoothing 41.22 ± 1.30. We ran every experiment requested in Q1–Q3.

W1 / Q1 — a controlled test of relation reversals. We ran it. It bounds the claim; it does not touch the results we report.

We scored each image against its true caption and the relation-reversed caption on 1,320 two-object left/right and above/below comparisons from 733 scenes (What'sUp controlled images, COCO and Visual Genome two-object sets). Every model is within three points of chance: template-only, balanced endpoints, ours and frozen OpenAI CLIP all fall between 49.9% and 52.9%.

This is the known floor of CLIP-style dual encoders on relation reversal (Kamath et al., 2023; Yuksekgonul et al., 2023, both cited in our introduction), and it has a data-side cause that changing the text target does not remove. In-batch negatives are captions of other images, so the contrastive loss is never asked to separate a caption from its relation-reversed twin; and the controlled test sets that exist are small enough that their intervals are about ±2 points wide, which cannot resolve differences between fine-tuned variants.

We therefore do not claim relation-reversal sensitivity. What the paper claims and measures is counting, distance and fine-grained local recognition, and there the effect is large and stable across seeds (Table R1): CLEVR-Counts 16.66 → 26.86 (+10.2), GTSRB 35.63 → 53.46 (+17.8), SVHN +7.2, CLEVR-Distance +1.4, and TallyQA +5.0 downstream under a matched LLaVA protocol (Table 2). In the revision we will state the relation-reversal result in Limitations.

W2 / Q2 — relational content, image-specific targets, or midpoint supervision? Table R3 separates the three.

All rows below use the identical arc law, Beta(0.2,0.2). Interior supervision and the presence of an image-specific caption signal in every batch are therefore held fixed; only the caption content changes.

Captions (same arc law)Spatial meanCLEVR-CountsGTSRBZS-11Flickr T→I
Template-only (no captions)32.06 ± 1.0116.66 ± 1.2835.63 ± 3.5269.04 ± 0.2066.08 ± 0.24
CapRL, neutral prompt (generic)33.66 ± 0.6020.31 ± 1.3838.70 ± 0.2066.35 ± 0.1461.40 ± 0.40
CapRL spatial, relation terms removed36.2720.8642.4666.5363.08
CapRL spatial (ours)41.22 ± 1.3026.86 ± 1.8653.46 ± 0.5169.62 ± 0.2267.12 ± 0.52

From Table R3 (ViT-L/14, %). ±: standard deviation over training seeds; rows without ± are single runs.

(1) Image-specific targets alone do not explain the gain. Generic captions from the same captioner carry an image-specific signal in every batch under the same law and add only +1.6 points. This also answers the instance-discrimination concern: that mechanism is equally available to the generic run.

(2) Relational content matters. Deleting only the relation terms from the same captions costs 5.0 of the 9.2 points (6.0 on CLEVR-Counts, 11.0 on GTSRB). Object names, count words and colour words are unchanged by the deletion. The relation-removed set is, by construction, the attribute-focused comparator the reviewer asks for: same objects and attributes, no relations. We used deletion rather than swapping because a swapped caption is false for its image, which confounds relational content with label noise.

(3) Midpoint supervision alone is not sufficient either. It is identical in all three arc rows. The gain needs both ingredients: interior targets (arc vs. balanced endpoints, +4.3 with identical captions, Table R2) and spatial content (+7.6 over generic captions under the same law).

(response continued in the next comment)

W3 — Proposition 2 is read through a leading-order term, and Figure 2 uses ImageNet. The identity is exact, and the sweep now covers the spatial metrics.

Eq. (9) is an exact identity for every symmetric law and needs no temperature assumption. Only the reading of the endpoint-slope term as midpoint supervision is leading-order, and the paper says so in Section 5.2 and in "Scope of the expansion" (Appendix D); we do not use it to predict accuracy. What the analysis does predict is the pair of hypotheses in Section 5.3, and Table R2 now tests both on the metrics that motivate the method:

Mixing lawInterior budgetImageNetSpatial meanCLEVR-CountsCLEVR-Dist.
Balanced endpoints086.23 ± 0.0136.92 ± 1.0021.73 ± 0.9724.66 ± 0.21
Beta(0.1,0.1)0.05885.9837.7424.4824.79
Beta(0.2,0.2)0.10185.53 ± 0.0341.22 ± 1.3026.86 ± 1.8625.99 ± 0.42
Beta(0.5,0.5)0.18281.2436.0822.2725.38
Beta(1,1), uniform0.25075.2331.9519.6923.57

From Table R2 (ViT-L/14, %, same captions and exposure in every row). Interior budget from Appendix Table 7.

Fidelity side: ImageNet falls monotonically as the interior budget grows (86.23 → 75.23), as predicted. Interior side: with captions and exposure fixed, switching on the interior term raises spatial accuracy by 4.3 points, and the gain is lost again once too much fidelity is spent (36.08 at Beta(0.5,0.5), 31.95 for the uniform law). Spatial accuracy is unimodal along the Beta path with its maximum at the paper's operating point. This is the trade-off of Section 5.3 measured on spatial tasks, and we will add these columns to Figure 2.

Q3 — CLEVR across the sweep; is a small shift off the endpoints enough? No. The gain tracks interior mass.

CLEVR-Counts and CLEVR-Distance are in the table above, and both peak at Beta(0.2,0.2). Beta(0.1,0.1) is the tested law closest to a distribution concentrated just inside the endpoints: 81% of its draws lie within 0.1 of an endpoint (central mass 18.7%, Appendix Table 7), and every batch already carries some caption-specific signal. It recovers about half of the CLEVR-Counts gain (21.73 → 24.48 → 26.86) but leaves the four-task mean at the endpoint level (37.74 vs 36.92), 3.5 points below Beta(0.2,0.2). Moving off the endpoints is therefore not sufficient: the effect grows with interior mass up to the operating point and reverses beyond it.

We hope these results resolve W1–W3 and Q1–Q3, and we would be grateful if the reviewer would consider raising the score.

Thanks for the rebuttal, which has addressed my concerns. I have no more questions.

We are deeply grateful to the reviewer for taking the time to read our rebuttal and for confirming that their concerns have been addressed. We truly appreciate the thoughtful feedback, which has helped strengthen our work, and we will make sure to include all requested analyses in the revised paper.

If the reviewer feels that our responses have satisfactorily resolved the points raised in the initial review, we would sincerely appreciate it if this could be reflected in the final rating. Thank you again for your valuable time and constructive insights. We remain happy to answer any further questions during the remainder of the discussion period.

Sincerely,

Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the three reviewers for their careful reading. The reviews ask for four things: multi-seed validation, the mixing-law sweep on spatial metrics, controls on caption content and on the objective, and a frozen-text-encoder ablation. We ran all of them. Every new run uses the recipe of Section 6 (ViT-L/14, same optimiser, schedule and evaluation). ± is the sample standard deviation over independent training seeds; entries without ± are single runs. Full results are in Tables R1–R4 in the comment below.

Seeds (Table R1). Over four seeds each, arc smoothing raises the four-task spatial zero-shot mean from 32.06 ± 1.01 (template-only) to 41.22 ± 1.30, with gains on all four tasks, +1.0 / +1.4 text-to-image R@1 on Flickr30K / MSCOCO, and ZS-11 transfer preserved (69.62 ± 0.22 vs 69.04 ± 0.20).

Mixing law on spatial metrics (Table R2). Spatial accuracy peaks at the paper's operating point: 36.92 for balanced endpoints, 37.74 at Beta(0.1,0.1), 41.22 at Beta(0.2,0.2), 36.08 at Beta(0.5,0.5) and 31.95 for the uniform law. ImageNet decreases monotonically along the same path, as Section 5.3 predicts.

Caption content (Table R3). Under the identical arc law, generic captions from the same captioner reach 33.66 ± 0.60 and relation-removed captions 36.27, against 41.22 for spatial captions. The gain comes from spatial content, not from additional caption text.

Objective (Table R3). Optimising both captions as simultaneous positives, without interpolation, is 12.5 spatial points below arc smoothing.

Frozen text encoder (Table R4). With only the image encoder trained, the counting gain is retained in full, but only about a quarter of the four-task gain survives and retrieval and transfer fall. The text encoder has to be trained, which is the premise of the method.

We will add Tables R1 and R2 to Section 7 and Tables R3–R4 to Appendix B.

New results: Tables R1–R4

± is the sample standard deviation over independent training seeds; entries without ± are single runs.

Table R1. Independent training seeds (ViT-L/14, top-1 %, R@1 %).

ModelSeedsSVHNGTSRBCLEVR-Dist.CLEVR-CountsSpatial tasks meanImageNetZS-11Flickr T→ICOCO T→I
Template-only451.40 ± 1.5235.63 ± 3.5224.55 ± 0.0316.66 ± 1.2832.06 ± 1.0186.39 ± 0.0569.04 ± 0.2066.08 ± 0.2439.32 ± 0.22
Arc, Beta(0.2,0.2), CapRL captions458.58 ± 2.6353.46 ± 0.5125.99 ± 0.4226.86 ± 1.8641.22 ± 1.3085.53 ± 0.0369.62 ± 0.2267.12 ± 0.5240.68 ± 0.05

Table R2. Mixing law, CapRL spatial captions (ViT-L/14).

ModelSVHNGTSRBCLEVR-Dist.CLEVR-CountsSpatial tasks meanImageNetZS-11Flickr T→ICOCO T→I
Spatial-caption-only (λ = 0)48.57 ± 5.2844.42 ± 6.9725.80 ± 0.7524.60 ± 1.1835.85 ± 3.2577.58 ± 0.0662.61 ± 0.3267.13 ± 0.9840.73 ± 0.04
Balanced endpoints (λ ∈ {0,1})55.30 ± 1.1345.98 ± 3.7124.66 ± 0.2121.73 ± 0.9736.92 ± 1.0086.23 ± 0.0166.38 ± 0.1466.69 ± 0.3039.61 ± 0.36
Arc, Beta(0.1,0.1)55.1446.5524.7924.4837.7485.9866.6159.7832.29
Arc, Beta(0.2,0.2)58.58 ± 2.6353.46 ± 0.5125.99 ± 0.4226.86 ± 1.8641.22 ± 1.3085.53 ± 0.0369.62 ± 0.2267.12 ± 0.5240.68 ± 0.05
Arc, Beta(0.5,0.5)54.7741.8925.3822.2736.0881.2466.0058.5831.37
Arc, Beta(1,1)56.4728.0823.5719.6931.9575.2363.4153.7028.84
Template-only (reference, no captions)51.40 ± 1.5235.63 ± 3.5224.55 ± 0.0316.66 ± 1.2832.06 ± 1.0186.39 ± 0.0569.04 ± 0.2066.08 ± 0.2439.32 ± 0.22

Table R3. Caption content and objective controls (ViT-L/14).

ModelSVHNGTSRBCLEVR-Dist.CLEVR-CountsSpatial tasks meanImageNetZS-11Flickr T→ICOCO T→I
Arc, CapRL spatial captions58.58 ± 2.6353.46 ± 0.5125.99 ± 0.4226.86 ± 1.8641.22 ± 1.3085.53 ± 0.0369.62 ± 0.2267.12 ± 0.5240.68 ± 0.05
Arc, generic captions51.23 ± 0.9938.70 ± 0.2024.42 ± 0.1620.31 ± 1.3833.66 ± 0.6085.69 ± 0.1266.35 ± 0.1461.40 ± 0.4034.63 ± 0.10
Arc, relation-removed captions57.4642.4624.2920.8636.2785.7566.5363.0834.27
Dual objective, no interpolation52.6917.2225.2519.8928.7686.2666.0956.6629.52
Balanced endpoints55.30 ± 1.1345.98 ± 3.7124.66 ± 0.2121.73 ± 0.9736.92 ± 1.0086.23 ± 0.0166.38 ± 0.1466.69 ± 0.3039.61 ± 0.36

Table R4. Frozen text encoder: only the image encoder is fine-tuned (ViT-L/14).

ModelSVHNGTSRBCLEVR-Dist.CLEVR-CountsSpatial tasks meanImageNetZS-11Flickr T→ICOCO T→I
Template-only, both towers trained51.40 ± 1.5235.63 ± 3.5224.55 ± 0.0316.66 ± 1.2832.06 ± 1.0186.39 ± 0.0569.04 ± 0.2066.08 ± 0.2439.32 ± 0.22
Arc, both towers trained58.58 ± 2.6353.46 ± 0.5125.99 ± 0.4226.86 ± 1.8641.22 ± 1.3085.53 ± 0.0369.62 ± 0.2267.12 ± 0.5240.68 ± 0.05
Template-only, text encoder frozen51.6150.7615.1014.9333.1085.7367.3470.4640.19
Arc, text encoder frozen50.1143.5620.8527.2935.4585.4865.6559.5629.90

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.