acceptodds
Under review as a conference paper at ICLR 2027

Attenuation or Removal? Calibrated Reading of Usable Conditional Information

Abstract

Cross-modal alignment trains a representation to match a description that is almost never complete. Does the alignment remove content the description omits, or merely make it harder to access? A finite probe cannot tell from a single operating point: its reading depends both on representation change and on where the target sits along its resolution curve. We separate representation-level conditional information from finite-resolution accessibility, and read usable information across Gaussian-noise resolution . Similarity transforms translate the resulting curve along , which is why the raw magnitude of a fixed-resolution diagnostic does not identify mechanism. Calibrating the probe with known invertible transforms then defines information-preserving nulls with an explicit floor and power region: a trajectory landing on such a null is not thereby shown invertible; it belongs to the instrument's observational equivalence class of information-preserving change. On fine-tuned CLIP checkpoints the Stanford Cars category channel contracts by nats of log-resolution while a construction-independent control does not, yet the trajectory remains indistinguishable from information-preserving attenuation; on CUB both targets expand instead, so we claim no universal sign. A local model of contrastive fine-tuning explains how this arises: alignment leaves the covariance-private block unchanged whereas the negative-sampling term contracts it, with an exactly preserving scalar regime and a perturbation bound around it. Synthetic ground truth separates attenuation from genuine removal, temperature interventions move contraction as predicted, and the calibrated pattern extends to DeepFashion, RSICD and COCO.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multimodal Representation Learning, Cross-Modal Alignment, Information Accessibility, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oKdzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether cross-modal alignment removes information omitted by text or makes it harder to access. The authors evaluate probes at different Gaussian noise levels and calibrate the results using known invertible transformations. Experiments on fine-tuned CLIP report target-dependent changes consistent with information-preserving attenuation. A local linear analysis examines how the contrastive objective can produce this effect.

Critical strengths
  1. The distinction between information loss and reduced probe accessibility is meaningful. Calibration with known invertible transformations provides a useful control for interpreting changes in probe performance. The authors also acknowledge that matching a preserving transformation does not prove information preservation.
  2. The Stanford Cars experiments include an injected control and three independent fine-tuning seeds. The category–control separation is consistent across these runs. Reporting the different trends on Cars and CUB also helps clarify the scope of the findings.
Critical weaknesses
  1. The reference used for calibration is unclear. Table 2 measures changes from epoch 0 to 30, while Table 5 and Appendix L construct calibration on epoch-30 embeddings. The change in L at a finite resolution depends on the starting curve, so matching the horizontal shift alone may not justify this comparison. This issue could affect the main placement result. In addition, Table 4 assigns zero change in L to a similarity, which appears inconsistent with Equation (4) and the nonzero changes reported in Table 5.
  2. The assumptions required for Equation (6) are not fully stated. The solution treats C_T/τ² as constant, but C_T depends on the trainable text head, and the experimental setup uses joint fine-tuning with a learnable temperature. The invariance assumption on the private subspace does not resolve this issue. Please clarify whether these quantities are frozen in the analysis and how this affects Theorems 1–2. The temperature experiment supports the qualitative trend, but does not establish that this approximation holds.
Questions for authors
  1. How is calibration on epoch-30 embeddings used to evaluate a trajectory starting at epoch 0? Would using epoch-0 embeddings as the calibration reference change the reported residuals?
  2. Please provide the isotropic–directed margins for RSICD and COCO. Do these margins exceed the reported floor of 0.14 sufficiently to distinguish the two calibration families?
3 comments
AuthorsOpenReview ↗

Response on Calibration Reference, Calibration Power, and Dynamics

We thank the reviewer for the precise technical comments on the calibration reference, the similarity signature, and the assumptions of the dynamical analysis. In particular, the reference-point concern affects the interpretation of our main Cars result; we have corrected the forward comparison and clarified the theoretical conditions.

1. Does using epoch 0 as the calibration reference change the result? (Weakness 1; Question 1)

Yes. It changes the residuals and reverses the tested-family conclusion for Cars category. The measured trajectory is Z0→Z30Z_0\to Z_{30}, so preserving calibration transformations for that forward comparison must act on the starting representation Z0Z_0. The revised calibration grids interpolate predictions at the measured shift Δ^\widehat\Delta, not at log⁡c\log c (Section 5; Appendix D, Tables 5–6).

The originally reported terminal-anchored comparison placed Cars category within the isotropic tolerance (residual −0.024-0.024 nats). Under the corrected forward comparison, the isotropic and directed residuals are −0.587-0.587 and −0.506-0.506 nats, respectively, both beyond the 0.0540.054-nat probe-refit floor (Table 2; Figure 3, p. 9). The same outcome holds in all three independent fine-tuning runs (Appendix K.2). Thus, the revised result lies outside both tested preserving families, whereas the submitted result appeared compatible with the isotropic family. The calibration families are not exhaustive, so this is not evidence establishing information removal.

Separately, the observed Cars reading has been corrected. At the paper's specified finest resolution, σmin⁡=0.0125\sigma_{\min}=0.0125, the recomputed endpoints are LY(0)=3.630L_Y(0)=3.630 and LY(30)=2.945L_Y(30)=2.945, giving ΔLobs=−0.686\Delta L_{\rm obs}=-0.686 nats. The submitted −0.472-0.472 was a fixed input in the figure-generation code and could not be matched to an available endpoint-probe run. We therefore use the readout computed directly from the cached representations under the stated protocol. This correction of ΔLobs\Delta L_{\rm obs} is distinct from re-anchoring the preserving calibration.

Appendix K.4 also gives the reverse Z30→Z0Z_{30}\to Z_0 expansion diagnostic. The category residual is +0.542+0.542 nats to either family; the two reverse predictions coincide at the matched shift, so this check does not distinguish the families and is not a second independent replication.

2. RSICD and COCO margins (Question 2)

No: the margins in the original terminal-anchored screens did not exceed the reported 0.140.14-nat floor. For every screened RSICD and COCO target, the reported isotropic–directed margin was at most 0.0680.068 nats. These calculations are terminal-anchored screening comparisons, not forward Z0Z_0 placements of the kind used for Cars/CUB. Accordingly, the revised appendix tables retain the observed target-wise Δ\Delta and ΔL\Delta L but remove isotropic-family assignments. We do not claim that those screening margins establish a mechanism or substitute for a new forward calibration on these corpora.

3. Similarity and the assumptions behind Equation (6) (Weaknesses 1–2)

Table 4. We corrected the similarity row: a similarity translates the resolution curve and generally changes the fixed-resolution probe reading. It therefore has ΔLY=ΔLiso(ΔY)\Delta L_Y=\Delta L_{\rm iso}(\Delta_Y), rather than zero, consistently with the nonzero calibration readings.

Dynamics. The instantaneous private-block equation permits time-varying A(t)=CT(t)/τ(t)2A(t)=C_T(t)/\tau(t)^2; the paired-alignment source vanishes on the covariance-private subspace instantaneously under the stated assumptions. The matrix-exponential closed form and near-isotropic perturbation bound require constant coefficients. The scalar rank-preserving regime also extends to A(t)=α(t)IA(t)=\alpha(t)I with fixed BB, giving WP(t)=WP(0)exp⁡ ⁣[−B∫0tα(s) ds].W_P(t)=W_P(0)\exp\!\left[-B\int_0^t\alpha(s)\,ds\right]. Our CLIP experiments jointly update both towers and temperature. We therefore interpret the constant-coefficient theory as a local model, and the temperature intervention as a qualitative directional check rather than verification of that approximation.

Summary

Correcting the forward calibration changes the Cars result from apparent isotropic compatibility to departure from both tested preserving families. The revised theory states its coefficient assumptions, and the cross-corpus screens no longer make family assignments unsupported by their resolution.

Reviewer_oKdzOpenReview ↗

My rating remains unchanged

The corrections to the similarity signature and the assumptions in Equation (6) address those concerns. However, correcting the calibration reference changes the main Cars conclusion, and the original endpoint change could not be matched to an available probe run. Therefore, I remain concerned about the reliability of the empirical results. The RSICD and COCO screens also no longer support the original calibration-family assignments. These issues affect the main empirical conclusions, so I retain my rating of 2 (Weak rejection).

AuthorsOpenReview ↗

Additional Calibration-Grid Validation of the Cars Endpoint Measurement

Thank you for your continued consideration. We performed an additional calibration-grid analysis to address the interpolation issue in the per-probe validation.

Additional calibration measurements. We retained the original calibration protocol and separately documented a post-hoc isotropic grid extension. Specifically, we added measurements at c=0.40c=0.40 and 0.450.45, covering seven noise levels and all five probe seeds. This added 70 calibration-grid cells to the original 280, without modifying the locked protocol. The evaluation used neval=2,011n_{\mathrm{eval}}=2,011 samples.

Probe-level results. With the extended isotropic grid, all five probe-specific shifts fall within the measured calibration range. The resulting paired residuals are:

ProbeΔ^\widehat\DeltaIso predictionIso residualIso statusDir predictionDir residualDir status
0−0.825−0.112−0.538Interp.−0.186−0.464Extrap.
1−0.600−0.101−0.578Interp.−0.160−0.519Interp.
2−0.800−0.192−0.529Interp.−0.266−0.455Extrap.
3−0.600−0.117−0.560Interp.−0.243−0.434Interp.
4−0.550−0.070−0.633Interp.−0.126−0.577Interp.
Mean−0.675—−0.5685/5—−0.4903/5
Sample SD0.127—0.041——0.058—

Residuals are computed as the observed endpoint shift minus the corresponding calibration-family prediction, evaluated at each probe's own estimated Δ^\widehat\Delta. All values are in nats except Δ^\widehat\Delta.

Importantly, the two isotropic residuals that previously required extrapolation remain substantially negative after interpolation using the additional measured points. The paper's mean-curve estimate is unchanged: Δ^=−0.600\widehat\Delta=-0.600, with an isotropic residual of −0.587-0.587 nats.

The directed calibration family was not extended. Three of its five probe-specific shifts interpolate and remain substantially separated from that family, while two remain outside its measured grid.

As an additional representation check, we re-extracted the initial representation Z0Z_0 using the official CLIP implementation and obtained element-wise agreement with the cached representation. The Z30Z_{30} model checkpoint was not available for an independent re-extraction.

Taken together, these results provide a probe-level robustness check for the revised Cars endpoint on the existing checkpoint pair. The isotropic result is consistent across all five probe refits and does not depend on calibration-grid extrapolation after the documented extension. This analysis does not constitute independent fine-tuning replications, a complete calibration confidence interval, or evidence of information removal.

Reviewer_fwL7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a method to discern information removal from information attenuation within embedding spaces. Their method relies on the addition of noise at several levels, and measures how much information a probe can extract at each level, which allows them to measure a curve of the ease of information access vs noise level. They calibrate this curve against known information-preserving transforms and compare to fine-tuning trajectories. Using this method, they find no evidence that categorical information is removed from the embedding space due to fine tuning. They do not find the same effect on the CUB dataset, so the effects appear to be dataset dependent.

Critical strengths

The question the authors are investigating is timely and it is good to see work in this direction. The experimental set up seems reasonable as it is stated.

Critical weaknesses

This manuscript is very dense and difficult to parse. I strongly urge the authors to make a pass for clarity. I also urge the authors make clear their contribution to the literature: the paper combines a measurement framework, a critique of single-resolution analyses, a theory of contrastive fine-tuning, and an empirical study, and it is not evident to me which is the main contribution or how each advances prior work.

One cannot replicate the work given the paper as it stands. I would urge the authors to release (anonymized at this stage) code and datasets for this.

The mixed results are worrying. Why do the Cars and CUB datasets act differently? I encourage the authors to explore this.

Questions for authors

What do the authors see as the main contribution, and how does it go beyond prior probing work such as control tasks?

Why do you think the results are so mixed for your method?

1 comment
AuthorsOpenReview ↗

Response on the Main Contribution, Mixed Results, and Reproducibility

We appreciate the reviewer's recognition of the importance of the question and the constructive suggestions on contribution, reproducibility, and the contrasting Cars/CUB results. We have clarified the calibrated measurement as the main contribution and made the scope of the empirical interpretation more explicit.

1. What is the contribution beyond control-task probing? (Critical Weakness; Question 1)

The primary contribution is a calibrated measurement of finite-resolution conditional accessibility. We estimate target-indexed resolution curves, measure their shifts Δ^\widehat\Delta, and compare the observed fixed-resolution readout change with the responses of known invertible transformations applied to the starting representation. Isotropic and target-directed preserving families provide explicit null comparisons, residuals, and a probe-refit floor (Sections 3–5; Table 2).

This differs from control tasks: random-label controls test whether a probe can fit an unrelated target; our calibration asks how the same real target reads after transformations known to preserve I(Y;Z∣G)I(Y;Z\mid G). The contrastive-dynamics model provides a possible explanation, while the checkpoint experiments test the behavior and limitations of this measurement. The revised Introduction makes this hierarchy explicit.

2. Why are Cars and CUB different? (Critical Weakness; Question 2)

The difference is present in the measured shifts, but we do not have a uniquely identified causal explanation. Cars category contracts by Δ^=−0.600\widehat\Delta=-0.600 log-resolution units and changes by ΔL=−0.686\Delta L=-0.686 nats. Against forward preserving calibrations its isotropic and directed residuals are −0.587-0.587 and −0.506-0.506 nats, exceeding the 0.0540.054-nat probe-refit floor. This departure from both tested families is reproduced in all three independent fine-tuning runs (Table 2; Figure 3; Appendix K.2).

CUB category instead expands by +0.325+0.325 log-resolution units. Its isotropic–directed margin is 0.0190.019 nats against a 0.0330.033-nat floor, so the instrument has insufficient power to distinguish those families there. Cars injected is near the intersection of the two families; CUB injected lacks an interpolated directed prediction. The data therefore differ both in shift direction and calibration power.

The local model allows target-dependent outcomes because its dynamics depend on cross-modal covariance rather than a universal contraction law. The datasets also have different caption regimes: Cars uses BLIP-generated descriptions, whereas CUB uses human-written attributes omitting species names (Appendix B). These are plausible factors to investigate, not established causes of the sign difference.

3. Reproducibility (Critical Weakness)

Appendix B now details the backbone, partitions, text reference, noise grid, standardization, probe and fitting rules, calibration transforms, and seed structure. The forward calibration grids appear in Appendix D, and training-run results in Appendix K.2.

We agree that an anonymized runnable code release would strengthen the work. Such a release has not yet been provided, and we do not claim that the written protocol alone replaces executable code.

Summary

The revision centers the calibrated instrument as the main contribution, reports the Cars/CUB contrast without assigning an untested cause, and makes the experimental protocol more explicit.

Reviewer_QiEAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Representation alignment aligns with a description that is mostly incomplete. A natural question arises: what happens to the unspecified content? For finite probes the paper defines four levels of separation between probe determined and representation changes. The representation level separates preservation from removal. The accessibility level tests finite resolution readout. The Geometric level sweeps the resolution and turns changes into translations and deformation of target-indexed curves. The method proposes calibration against known invertible transforms and identifies observational equivalence classes.

Critical strengths

The idea itself is interesting and the problem is important. Alignment is a standard technique in multi-modal modeling and it is of interest to know whether information not common to both representation is preserved, attenuated or removed.

Critical weaknesses

The paper is unclear and poorly written. It is hard to read, and uses dense and obscure language. It is already hard to tell from the abstract what the paper is doing.

Section 3 immediately starts without setting notation or describing the mathematical setting, leaving the reader guessing as to what the variables mean.

Definition 1 uses V_Q under a measurement procedure but does not define what it is. It appears to be a basic quantity and does not appear to be defined.

The ‘Scope and Conclusion’ section is similarly poorly written and it remains unclear what the high-level conclusions and limitations of the work are being expressed.

The paper does not clarify its new contributions: A list in the introduction would be useful.

Questions for authors

The first paragraph says improved alignment could exist with ‘weaker access’. can you clarify what this means?

What do you mean by ‘finite resolution accessibility’?

this paragraph: accessibility level ‘finite resolution readout may weaken even that quantity if unchanged’ explain this.

What are representations Z_0 and Z_t meant to indicate in definition 1?

2 comments
AuthorsOpenReview ↗

Response on Presentation, Definitions, and the Main Conclusion

We thank the reviewer for identifying where the presentation obscured the central question and its mathematical setting. We agree that the notation, the finite-resolution readout, and the distinction from representation-level information must be clear before the formal claims. We have revised the Introduction, Section 3, and the Conclusion accordingly.

1. Mathematical setting and VQV_{\mathcal Q} (Critical Weaknesses; Question 4)

Section 3 (pp. 2–3) now introduces XX (image), TT (description), YY (target), G=g(T)G=g(T) (frozen text reference), and Zt=ft(X)Z_t=f_t(X) (visual representation at fine-tuning checkpoint tt). Z0Z_0 and ZtZ_t represent the same image at different checkpoints. The training index tt is distinct from the resolution coordinate u=log⁡σu=\log\sigma.

We define the operational readout before Definition 1:

\widehat R_{\rm test}(\widehat q_G) -\widehat R_{\rm test}(\widehat q_{ZG}),$$ the held-out log-loss improvement from adding the noisy visual representation to a text-only probe. Section 3 explains fitting, fallback, and the live resolution band; Appendix B specifies the probe configuration. ### 2. “Weaker access” and “finite-resolution accessibility” (Questions 1–3) *Weaker access* means a smaller reading from a specified finite-capacity probe at a specified Gaussian-noise resolution. This is not a decrease in the representation-level conditional mutual information $I(Y;Z\mid G)$. Section 3 now uses an explicit example: an invertible rescaling $Z\mapsto cZ$ preserves $I(Y;Z\mid G)$ but translates the resolution curve, potentially lowering $V_{\mathcal Q}$ at a fixed $\sigma$. The earlier statement that a finite-resolution reading might “weaken” an unchanged information quantity was unclear. **The probe reading can weaken while the underlying conditional mutual information remains unchanged.** This distinction motivates the need for calibration rather than interpreting a single-resolution score as removal. ### 3. Contribution and scope (Critical Weaknesses) The Introduction (pp. 1–2) now identifies **calibrated resolution-curve measurement** as the primary contribution. The four-level roadmap distinguishes representation information, probe accessibility, resolution-curve geometry, and comparison with known invertible transformations. It also explains why this differs from control-task probing, which evaluates artificial-label learning rather than the effect of a known information-preserving map. The revised conclusion (p. 10) states the findings directly: Cars category accessibility contracts beyond the two tested preserving families across three training runs; CUB expands with limited family discrimination. These are claims about an operational measurement, not proof of information removal. The conclusion also spells out the single-backbone setting, local dynamical assumptions, and unresolved sufficiency of the frozen text reference $G$ for $T$. These revisions aim to make the measured object and the limits of its interpretation understandable on a first reading.

Thank you for the response. However, I believe this paper requires a significant rewrite. I will maintain my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to the Reviewers

Dear Area Chair and Reviewers,

We sincerely thank all three reviewers for their careful readings and constructive feedback. The reviews highlighted two important weaknesses in the submitted paper: the measurement framework was difficult to follow, and the reference used for the principal calibration comparison was not aligned with the forward training trajectory. We have revised the manuscript to address both.

Presentation and contribution. The Introduction now identifies calibrated resolution-curve measurement as the central contribution and distinguishes it from control-task probing. Section 3 introduces the notation and finite-probe readout before the definitions, and the Conclusion states the empirical findings, assumptions, and limitations more directly.

Calibration and empirical interpretation. For the Z0→Z30Z_0\to Z_{30} trajectory, we now construct the two information-preserving calibration families on Z0Z_0. We also recompute the observed fixed-resolution endpoint readout under the stated measurement definition. The corrected Cars category trajectory lies outside both tested preserving families across three fine-tuning runs, rather than appearing compatible with the isotropic family. This does not establish information removal. CUB instead shows expansion with limited power to distinguish the calibration families.

Theory and reporting. We corrected the similarity signature at fixed resolution, clarified which dynamical claims require constant coefficients, and distinguished the primary forward-calibration findings from the descriptive cross-corpus screens. The revised manuscript marks the most substantive changes in blue. We address each reviewer's specific concerns and questions below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.