- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether cross-modal alignment removes information omitted by text or makes it harder to access. The authors evaluate probes at different Gaussian noise levels and calibrate the results using known invertible transformations. Experiments on fine-tuned CLIP report target-dependent changes consistent with information-preserving attenuation. A local linear analysis examines how the contrastive objective can produce this effect.
Critical strengths
- The distinction between information loss and reduced probe accessibility is meaningful. Calibration with known invertible transformations provides a useful control for interpreting changes in probe performance. The authors also acknowledge that matching a preserving transformation does not prove information preservation.
- The Stanford Cars experiments include an injected control and three independent fine-tuning seeds. The category–control separation is consistent across these runs. Reporting the different trends on Cars and CUB also helps clarify the scope of the findings.
Critical weaknesses
- The reference used for calibration is unclear. Table 2 measures changes from epoch 0 to 30, while Table 5 and Appendix L construct calibration on epoch-30 embeddings. The change in L at a finite resolution depends on the starting curve, so matching the horizontal shift alone may not justify this comparison. This issue could affect the main placement result. In addition, Table 4 assigns zero change in L to a similarity, which appears inconsistent with Equation (4) and the nonzero changes reported in Table 5.
- The assumptions required for Equation (6) are not fully stated. The solution treats C_T/τ² as constant, but C_T depends on the trainable text head, and the experimental setup uses joint fine-tuning with a learnable temperature. The invariance assumption on the private subspace does not resolve this issue. Please clarify whether these quantities are frozen in the analysis and how this affects Theorems 1–2. The temperature experiment supports the qualitative trend, but does not establish that this approximation holds.
Questions for authors
- How is calibration on epoch-30 embeddings used to evaluate a trajectory starting at epoch 0? Would using epoch-0 embeddings as the calibration reference change the reported residuals?
- Please provide the isotropic–directed margins for RSICD and COCO. Do these margins exceed the reported floor of 0.14 sufficiently to distinguish the two calibration families?
3 comments
My rating remains unchanged
The corrections to the similarity signature and the assumptions in Equation (6) address those concerns. However, correcting the calibration reference changes the main Cars conclusion, and the original endpoint change could not be matched to an available probe run. Therefore, I remain concerned about the reliability of the empirical results. The RSICD and COCO screens also no longer support the original calibration-family assignments. These issues affect the main empirical conclusions, so I retain my rating of 2 (Weak rejection).
Additional Calibration-Grid Validation of the Cars Endpoint Measurement
Thank you for your continued consideration. We performed an additional calibration-grid analysis to address the interpolation issue in the per-probe validation.
Additional calibration measurements. We retained the original calibration protocol and separately documented a post-hoc isotropic grid extension. Specifically, we added measurements at and , covering seven noise levels and all five probe seeds. This added 70 calibration-grid cells to the original 280, without modifying the locked protocol. The evaluation used samples.
Probe-level results. With the extended isotropic grid, all five probe-specific shifts fall within the measured calibration range. The resulting paired residuals are:
| Probe | Iso prediction | Iso residual | Iso status | Dir prediction | Dir residual | Dir status | |
|---|---|---|---|---|---|---|---|
| 0 | −0.825 | −0.112 | −0.538 | Interp. | −0.186 | −0.464 | Extrap. |
| 1 | −0.600 | −0.101 | −0.578 | Interp. | −0.160 | −0.519 | Interp. |
| 2 | −0.800 | −0.192 | −0.529 | Interp. | −0.266 | −0.455 | Extrap. |
| 3 | −0.600 | −0.117 | −0.560 | Interp. | −0.243 | −0.434 | Interp. |
| 4 | −0.550 | −0.070 | −0.633 | Interp. | −0.126 | −0.577 | Interp. |
| Mean | −0.675 | — | −0.568 | 5/5 | — | −0.490 | 3/5 |
| Sample SD | 0.127 | — | 0.041 | — | — | 0.058 | — |
Residuals are computed as the observed endpoint shift minus the corresponding calibration-family prediction, evaluated at each probe's own estimated . All values are in nats except .
Importantly, the two isotropic residuals that previously required extrapolation remain substantially negative after interpolation using the additional measured points. The paper's mean-curve estimate is unchanged: , with an isotropic residual of nats.
The directed calibration family was not extended. Three of its five probe-specific shifts interpolate and remain substantially separated from that family, while two remain outside its measured grid.
As an additional representation check, we re-extracted the initial representation using the official CLIP implementation and obtained element-wise agreement with the cached representation. The model checkpoint was not available for an independent re-extraction.
Taken together, these results provide a probe-level robustness check for the revised Cars endpoint on the existing checkpoint pair. The isotropic result is consistent across all five probe refits and does not depend on calibration-grid extrapolation after the documented extension. This analysis does not constitute independent fine-tuning replications, a complete calibration confidence interval, or evidence of information removal.
Response on Calibration Reference, Calibration Power, and Dynamics
We thank the reviewer for the precise technical comments on the calibration reference, the similarity signature, and the assumptions of the dynamical analysis. In particular, the reference-point concern affects the interpretation of our main Cars result; we have corrected the forward comparison and clarified the theoretical conditions.
1. Does using epoch 0 as the calibration reference change the result? (Weakness 1; Question 1)
Yes. It changes the residuals and reverses the tested-family conclusion for Cars category. The measured trajectory is Z0→Z30, so preserving calibration transformations for that forward comparison must act on the starting representation Z0. The revised calibration grids interpolate predictions at the measured shift Δ, not at logc (Section 5; Appendix D, Tables 5–6).
The originally reported terminal-anchored comparison placed Cars category within the isotropic tolerance (residual −0.024 nats). Under the corrected forward comparison, the isotropic and directed residuals are −0.587 and −0.506 nats, respectively, both beyond the 0.054-nat probe-refit floor (Table 2; Figure 3, p. 9). The same outcome holds in all three independent fine-tuning runs (Appendix K.2). Thus, the revised result lies outside both tested preserving families, whereas the submitted result appeared compatible with the isotropic family. The calibration families are not exhaustive, so this is not evidence establishing information removal.
Separately, the observed Cars reading has been corrected. At the paper's specified finest resolution, σmin=0.0125, the recomputed endpoints are LY(0)=3.630 and LY(30)=2.945, giving ΔLobs=−0.686 nats. The submitted −0.472 was a fixed input in the figure-generation code and could not be matched to an available endpoint-probe run. We therefore use the readout computed directly from the cached representations under the stated protocol. This correction of ΔLobs is distinct from re-anchoring the preserving calibration.
Appendix K.4 also gives the reverse Z30→Z0 expansion diagnostic. The category residual is +0.542 nats to either family; the two reverse predictions coincide at the matched shift, so this check does not distinguish the families and is not a second independent replication.
2. RSICD and COCO margins (Question 2)
No: the margins in the original terminal-anchored screens did not exceed the reported 0.14-nat floor. For every screened RSICD and COCO target, the reported isotropic–directed margin was at most 0.068 nats. These calculations are terminal-anchored screening comparisons, not forward Z0 placements of the kind used for Cars/CUB. Accordingly, the revised appendix tables retain the observed target-wise Δ and ΔL but remove isotropic-family assignments. We do not claim that those screening margins establish a mechanism or substitute for a new forward calibration on these corpora.
3. Similarity and the assumptions behind Equation (6) (Weaknesses 1–2)
Table 4. We corrected the similarity row: a similarity translates the resolution curve and generally changes the fixed-resolution probe reading. It therefore has ΔLY=ΔLiso(ΔY), rather than zero, consistently with the nonzero calibration readings.
Dynamics. The instantaneous private-block equation permits time-varying A(t)=CT(t)/τ(t)2; the paired-alignment source vanishes on the covariance-private subspace instantaneously under the stated assumptions. The matrix-exponential closed form and near-isotropic perturbation bound require constant coefficients. The scalar rank-preserving regime also extends to A(t)=α(t)I with fixed B, giving WP(t)=WP(0)exp[−B∫0tα(s)ds]. Our CLIP experiments jointly update both towers and temperature. We therefore interpret the constant-coefficient theory as a local model, and the temperature intervention as a qualitative directional check rather than verification of that approximation.
Summary
Correcting the forward calibration changes the Cars result from apparent isotropic compatibility to departure from both tested preserving families. The revised theory states its coefficient assumptions, and the cross-corpus screens no longer make family assignments unsupported by their resolution.