- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper seems to attempt to measure how model's response changes by the perturbations of the input-caption in the contrastive image-text training. The authors seem to propose an approach to measure the change by fine-tuning the open-CLIP model on some downstream image classification datasets. They conduct experiments on several datasets and models and present some response analysis on the model's response change w.r.t the perturbation.
Critical strengths
- They seem to attempt to measure how the CLIP-like model's output changes by input variations. This kind of topic might be interesting to community.
Critical weaknesses
- I could not understand the scope of this paper due to the poor writing. Their contribution and problem setting are not clear at all from the abstract and introduction. The positioning of their paper is also not clear since they cite no papers in the introduction. They need major refinement w.r.t. the paper writing. For example, line 52, "We test this measurement by changing image-caption correspondence while holding the predictor and evaluation panel fixed.", what does the sentence mean?
- They do not mention the contribution considering the existing work. It is hard to measure the value of this paper.
- Their theory section is also hard to understand. What does "STATISTICAL TOKENS" mean? in the title of the Sec.3. The term is never mentioned after this title.
- Experiments sections are also poorly organized. They do not mention the goal of the experiments, detailed setting, and obtained insights.
- Their empirical results look very specific to the datasets. They do not mention any general observations from experiments.
Questions for authors
The authors need major refinement on this submission for better readability.
Response on Problem Setup, Contribution, and Empirical Interpretation
We appreciate the reviewer's emphasis on readability and on making the contribution distinguishable from existing work. We agree that readers should be able to identify the scientific question, the statistical object, and the purpose of each experiment before reaching the technical details. The revised manuscript reorganizes those entry points as follows.
1. Problem setting and Introduction (Critical Weakness 1)
The Abstract (p. 1) now states two questions: how to measure the relative response to specified input variations without dependence on embedding coordinates, and whether that response allocation depends on image–caption pairing. The Introduction (pp. 1–2) begins with a concrete car example, cites related work, and lists three contributions. The previously compressed sentence about “changing correspondence” is now explained operationally: correct-caption and image-reassigned-caption runs share the images, caption multiset, evaluation panel, and frozen predictor. The manipulated factor is their training correspondence.
2. Contribution relative to prior work (Critical Weakness 2)
Section 2 (pp. 2–3) and Appendix C (Table 7) compare the approach with global CKA/Procrustes analyses and per-cell adaptations of similarity measures. The contribution is a transformation-resolved, affine-invariant covariance reading with normalized shares, cross-family overlap, and an exact covariance budget (Theorem 1). We do not claim that no existing measure detects a pairing reversal: the stochastic shape metric in Table 7 does so as well. The new information provided here is the interpretable allocation across declared variations.
3. Meaning of statistical tokens and theory (Critical Weakness 3)
Section 3.1 (p. 3) now introduces the term before Definition 1. Repeatedly applying a fixed transformation to one image creates a distribution of embeddings, called its statistical token. Its within-anchor covariance is aggregated across images for a cell; the share is computed only relative to the whole mixture. Section 3.2 explains the readout before the formulas; Table 4 (p. 14) provides a notation guide. The conditional directional theory is shortened in Section 4, with supporting statements in Appendix A.
4. Experimental organization and takeaways (Critical Weakness 4)
The new experiment map in Section 6 (p. 6) states the purpose of each part: validating the readout (§6.1), measuring allocation (§6.2), testing pairing (§6.3), grading corruption (§6.4), testing caption-content attribution (§6.5), and relating responses to accuracy (§6.6). Appendix E records the training arms, seeds, transform pipeline, and evaluation protocol. The main controlled Cars comparison gives +0.833 versus −0.443 (Table 1), with the family-block inference in Table 2.
5. Cross-setting observations and scope (Critical Weakness 5)
Section 6.3 and Figure 2(b) report the pairing-sign contrast on Cars, CUB, Flowers, and a second public ViT initialization. ConvNeXt has the same observed signs but does not pass the declared CCA gate, so it is explicitly a sensitivity check. Section 7 (p. 10) synthesizes the recurring pattern and its limits: effect magnitudes are not directly comparable across separately calibrated settings and unequal optimization budgets; behavioral correlations are also not uniform across datasets.
Summary (Question for Authors: overall readability)
The revisions directly address the requested presentation refinement: a concrete opening, an earlier intuitive definition of the main object, explicit positioning against nearby methods, a shorter conditional theory, and an experiment-by-experiment rationale. The scientific scope remains unchanged. We hope this makes the method, empirical finding, and interpretation limits straightforward to assess.