acceptodds
Under review as a conference paper at ICLR 2027

A Unifying Framework for Concept-Based Representational Similarity

Abstract

Learned representations across models and modalities often exhibit striking structural similarities, suggesting shared underlying concept decompositions. However, concept alignment remains poorly defined: existing approaches optimize different objectives under the same terminology, obscuring what is actually aligned. We propose a unifying framework that decomposes alignment along two axes: *what* is aligned (representations vs. concepts) and *at what level* (instance-wise vs. distributional). This induces four corresponding properties - instance-wise and distributional variants of translation and concept consistency - and reveals precisely which of these guarantees existing methods provide. We further introduce InterVenchA, an intervention-based benchmark that separately measures extraction quality, translation quality, and concept consistency. Through theory and experiments, we show that commonly assumed equivalences between alignment objectives fail in practice: optimizing one property does not reliably recover the others, and purely unsupervised objectives fail to recover meaningful instance-level alignment. We then propose the Coupled Sparse Autoencoder (CoSAE), which jointly enforces complementary alignment objectives. Strong alignment emerges only in this regime. Surprisingly, as little as 0.1% paired data is sufficient to recover instance-level alignment when anchoring distributional objectives. Overall, our results show that concept alignment is fundamentally multi-objective: it must be defined, measured, and optimized as such.

Primary area:
interpretability and explainable AI
Keywords:
concept-based interpretability, interpretability, Sparse Autoencoder, SAE, crosscoder

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_EuUCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper studies the concept alignment problem, i.e., measuring the consistency of the underlying concepts across two arbitrary representations (e.g., embeddings from different modalities or different models). The authors argue that ideal concept alignment should possess four indispensable properties, i.e., translation, concept consistency, instance-wise alignment, and distributional alignment. Based on such observations, the authors propose a compositional training objective for training Sparse Autoencoders (SAEs). To evaluate the quality of the trained SAEs, the authors further introduce the InterVenchA benchmark, which extends the SAEBench to assess translation quality and concept consistency.

Critical strengths
  1. The theoretical analysis is solid. Based on the authors' discussion, using these four properties to evaluate concept agreement is reasonable, and each property is indispensable.

  2. Introducing InterVenchA is reasonable. Translation and concept consistency are more informative than a single representation-similarity score.

Critical weaknesses
  1. Due to the unclear presentation, the paper is difficult to follow. For example, 1) the symbols P_{dcy}, P_{tr}, P_{cont}, and P_{cy} that appear in Figure 1 are not explained in its caption or in the Introduction section. This confusion was not resolved until the Framework Formulation section. 2) The metrics in Table 1 should be annotated with whether higher values are better. 3) The regularization settings in Table 1 are also confusing. Variants (h)–(j) use paired data and specify the proportion, yet it is unclear how much paired data variants (b), (d), and (f) use, given that the L_{tr} and L_{count} losses are also based on paired data. 4) It is unclear why the results obtained with the full L_{SAE} are not reported in Table 1.

  2. The main experiments aim to illustrate the effect of different regularization schemes on the quality of the trained SAEs. However, the weights of the different loss terms vary dramatically, e,g., α_{tr}=1 while α_{tr}^{dist}=1000. Such a large discrepancy makes variants (b) and (c) not directly comparable. A fair comparison of their contributions should be based on the optimal weight for each loss term.

  3. In practice, applying Equation (1) to train the SAE raises the question of which regularization terms to retain and how to configure the weights, making the search space prohibitively large.

  4. The authors need to demonstrate that the metrics (e.g., M_{tr} and M_{cont}) can reliably and correctly rank the relative concept consistency of different model pairs, rather than merely showing that a particular training objective yields higher scores. For example, given an anchor model A and two comparison models B and C, one needs to determine which of them has higher concept alignment with A. What matters here is discriminability rather than absolute values. Assume SAE1<A, B>=0.8, SAE1<A, C>=0.7, SAE2<A, B>=0.6, and SAE2<A, C>=0.1. Although SAE2 yields lower scores overall, it provides a clearer interpretation.

  5. Does CoSAE refer to L_{SAE}? If so, why is Equation (1) not called L_{CoSAE}?

Questions for authors

Response to the questions in the weaknesses.

1 comment
AuthorsOpenReview ↗

Rebuttal to Reviewer EuUC

We thank the reviewer for a thorough and careful review. The detailed list of clarity issues is extremely useful, and makes it clear that the presentation needs work. We address every point below.


W1: Clarity.

  1. The Figure 1 caption will name each property (PdcyP_{\mathrm{dcy}}: demi-cycle reconstruction, PtrP_{\mathrm{tr}}: translation, PcontP_{\mathrm{cont}}: concept consistency, PcyP_{\mathrm{cy}}: full cycle), with a pointer to Section 3.
  2. All metrics are higher-is-better. Since Table 1 is already very wide, we will state this in the caption rather than adding arrows to each column.
  3. Lines (b), (d) and (f) use 100% paired data. We will state this explicitly in the table.
  4. The reviewer is right that the full-objective row is missing from Table 1. The CoSAE reported in Table 2 corresponds to this missing line, with scores very close to lines (h) and (i). We will add this row.

W2: Different weights across loss terms.

  • The weights are not set arbitrarily. Each α\alpha is calibrated independently with the same procedure (Appendix H.2): the largest order of magnitude that does not degrade self-reconstruction. In this sense, each term is compared at its own best setting under a common criterion.
  • The gap between αtr=1\alpha_{\mathrm{tr}} = 1 and αtrdist=103\alpha^{\mathrm{dist}}_{\mathrm{tr}} = 10^3 reflects gradient norms. The distributional losses produce much smaller gradients because of the low value of random projections in high dimensional spaces. These small gradients must be scaled up to have any effect relative to Ldcy\mathcal{L}_{\mathrm{dcy}}.
  • Results are stable when α\alpha varies by one to two orders of magnitude.
  • Gradient-normalized losses would arguably be more principled. We can add this to the appendix, but chose to report the calibrated α\alpha values directly to avoid adding another layer of complexity.

W3: Which terms to retain and how to configure them. We thank the reviewer for this comment. Identifying which terms matter and how to configure them is precisely one of the goals of the paper, but the reviewer's comment shows that we did not state this clearly enough. We will add the following sentence right after Equation (1):

"Combining this many terms is impractical as is. The remainder of this paper therefore investigates the precise effect of each regularization, in order to identify which ones are needed in practice and how to calibrate them."

W4: Discriminability across model pairs. This is a very interesting point, and we thank the reviewer for the discussion.

  • Our focus was deliberately on finer-grained analysis. The motivation for concept-based similarity is to go beyond single summary scores such as CCA or CKA, which remain the natural tools for ranking models globally.
  • MalignM_{\mathrm{align}} obtained from CoSAEs trained on different model pairs could serve as such a global score, but characterizing its ranking properties is beyond the scope of this paper.
  • The human study (Appendix D) provides evidence of discriminability at the feature level: atoms that MalignM_{\mathrm{align}} labels as well aligned are significantly preferred by annotators over poorly aligned ones.
  • We will add model-pair discriminability as a direction for future work.

W5: CoSAE vs. LSAE\mathcal{L}_{\mathrm{SAE}}. Yes, Equation (1) is the CoSAE objective. Other reviewers raised the same point, and we will rename it LCoSAE\mathcal{L}_{\mathrm{CoSAE}}. Thank you for flagging this.


We are grateful for the reviewer's time and careful reading. We believe these changes substantially improve the paper's readability. We hope our responses address the concerns, and that the reviewer will consider raising their score.

Reviewer_mu66OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors decompose representation alignment into translation and concept consistency, each at the instance-wise and distributional level, and motivate their complementary nature from a first-principles perspective. After introducing these concepts, the authors are able to design useful differentiable proxies, which are then used to define the loss of their proposed CoSAE. The empirical results reported seem to suggest that optimizing translation does not necessarily yield concept consistency and vice versa. Furthermore, the proposed training seems comparable to or better than existing approaches on useful benchmarks.

Critical strengths
  • Useful decomposition of representation alignment into translation and concept consistency, at both the instance-wise and distributional level.
  • Insightful connection between critical desiderata from a first-principles perspective, e.g. translation and consistency, and concrete instantiations that can be readily optimized to deliver strong performance in feature/concept interpretation use-cases.
  • Overall, well-executed work with high-level insights across the text, backed with strong empirical results in varied applications. In particular, I find the empirical results from Table 1 and Table 2 impressive.
Critical weaknesses
  • It is unclear what the loss of the authors' proposed CoSAE is: in its current form, (1) is written as \mathcal{L}{SAE}, which reads as the typical SAE loss. I believe a better alternative would be to define the loss of a typical SAE with \mathcal{L}{SAE} and then define the loss of the proposed CoSAE with \mathcal{L}_{CoSAE}.
  • Equation (6) only names an MMD. Appendix G specifies a sliced empirical-characteristic-function estimator, but the main text still does not make the projections, weighting, or batch computation concrete enough to reimplement.
  • In Table 1, the authors ablate the proposed loss components and evaluate them with InterVenchA. These scores adapt SAEBench metrics, and a human study supports the aggregate alignment score, but the results would be much more convincing if metrics proposed independently of this paper were also used to measure concept consistency and translation. Additionally, without reading the SAEBench paper it is not clear to me the difference between these two.
  • The loss proposed in (1) uses many more coefficients than a traditional SAE, which typically takes 1 or 2 hyperparameters. Appendix H.2 does give an order-of-magnitude calibration, but I still expect this to be a major limitation for practitioners aiming to use this approach.
  • I find that Figure 6 exaggerates the result. The comparison to CLIP trained from scratch is not fair, as CoSAE is leveraging pretrained models, so this is closer to distillation than to contrastive training from scratch.
  • Style preference: I find the extensive use of em-dashes and semicolons annoying to read.
Questions for authors

Could you provide a more concrete formal contextualization of how your loss relates to the vanilla SAE loss?

Is it possible to complement the results from Table 1 with independently proposed metrics? Or to clarify to what extent your existing metrics differ from the ones in SAEBench

A meaningful experiment/theoretical insights offering concrete guidance on the sensibility and selection of hyperparameters that addresses my concerns (see above) would improve my score.

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer mu66

We thank the reviewer for a careful, detailed and encouraging review, and for the concrete suggestions on presentation. We start with the hyperparameter question, which the reviewer flagged as most important, and then address the other points.

Hyperparameter sensitivity and selection. We do not have theoretical guarantees here, but we can offer empirical guidance.

Calibration rule (Appendix H.2, lines 1401–1406).

  • We increase each regularization strength until it starts to hurt self-reconstruction, and stop there. This ensures that the alignment terms act as regularizers and do not break concept discovery.
  • Ldcy\mathcal{L}_{\mathrm{dcy}} is the original SAE objective, not an alignment term, and its weight is fixed to 1 throughout.

Two regimes. For all alignment terms except Ltr\mathcal{L}_{\mathrm{tr}}, we observe a smooth transition between two regimes:

  • α\alpha too small: the gradients are negligible compared to those of self-reconstruction, and the regularization has no effect.
  • α\alpha too large: the regularization dominates and leads to degenerate solutions, where shared structure is hallucinated at the expense of self-reconstruction.
  • In between: genuine alignment is recovered. Results are stable over one to two orders of magnitude of α\alpha in most cases.

Ltr\mathcal{L}_{\mathrm{tr}} behaves differently.

  • Its role is close to that of Ldcy\mathcal{L}_{\mathrm{dcy}}, so it does not produce degenerate solutions.
  • When it dominates, however, the model stops doing self-reconstruction, focuses only on translation, and gets stuck in local minima.
  • Balancing the two lets the model recover meaningful structure and align it at the same time. This yields both better reconstruction R2R^2 and better translation R2R^2.

Practical configurations.

  • When all pairs are matched (e.g. unimodal settings), instance-wise terms are sufficient. This leaves only two hyperparameters, and αtr=αcont=1\alpha_{\mathrm{tr}} = \alpha_{\mathrm{cont}} = 1 worked across all our settings.
  • In mixed settings where distributional alignment matters, four hyperparameters remain. αdist=103\alpha^{\mathrm{dist}} = 10^3 worked across all our settings.

We will move this discussion from the appendix to the main body in our final revision, and detail it as above.


Other clarifications

Loss naming / relation to the vanilla SAE loss. We will rename Eq. (1) to LCoSAE\mathcal{L}_{\mathrm{CoSAE}}. The vanilla SAE loss is self-reconstruction, Ldcy\mathcal{L}_{\mathrm{dcy}}. Line 240 will read:

"The typical SAE loss is self-reconstruction [44], which corresponds to the property PdcyP_{\mathrm{dcy}}. As is standard practice, we operationalize it as an ℓ2\ell_2 reconstruction loss (Equation (2))."

Reimplementing Eq. (6).

  • As stated in line 266, the distributional loss follows [57]: a sliced empirical characteristic function test, which we show is equivalent to a sliced MMD (Proposition 3).
  • We will add a code snippet for this loss, to the appendix.
  • We will also add a link to the code repository in the final version.

Independent metrics and differences with SAEBench.

  • We agree that standard metrics would be preferable, and we will state this limitation explicitly. Unfortunately, existing alignment metrics are often case-specific and do not transfer to the general setting:
    • SAE-A defines metrics specific to the joint embedding space of CLIP;
    • [65] defines scores specific to the shared code of crosscoders.
  • Translation R2R^2 and sparse probing are already used in prior work (e.g. USAE). Our other metrics are built on top of sparse probing, following SAEBench.
  • The differences with SAEBench are twofold:
    1. SAEBench evaluates a single SAE and only probes in concept space. We add latent-space variants, where the probe is a sparse combination of dictionary atoms. This distinction is new: practitioners usually just report "sparse probing", although the two variants measure fundamentally different things.
    2. We turn each self-evaluation into a transfer evaluation, by fitting an intervention on domain ii and applying it on domain jj.
  • Our metrics are designed to be general, and we hope they can support cross-paper comparisons. The human study (Appendix D) provides an independent check for the alignment score.

Figure 6. We agree. The comparison to Radford et al. is included only for context. The meaningful comparison is with Maniparambil et al. [64], which also builds on frozen pretrained backbones, as stated in the main text. We will revise the figure and caption to make this the primary comparison.


We thank the reviewer again for their time and for the constructive suggestions, which will substantially improve the clarity and practical usefulness of the paper.

Thank you to the reviewers for their thorough and detailed comments.

In the future, I would recommend the authors to highlight the revised text with a different color to make it easier for the reviewers to analyze the manuscript changes.

Most of my concerns were minor and the authors do a good job at addressing them.

Regarding my main concern with respect to the large number of difficult-to-tune hyperparameters:

  • The authors offer a distinction between paired and unpaired settings, and state that for the paired setting the problem simplifies to tuning the instance-wise terms. This is a very useful distinction that streamlines the model's application in the paired setting.
  • The authors provide a detailed description of how to tune these hyperparameters, as well as a useful intuitive overview of why specific selections might lead to undesirable outcomes

Despite this, unfortunately, I do not remain fully convinced. Ideally, the authors would provide either a more formal exploration or a more ambitious experimental exploration. The current explanations are potentially correct, but in their current form they are not possible to meaningfully verify/validate, as such I find them handwavy and speculative. As a consequence, I do not feel confident that the proposed method is as equally useful for the community as simpler frameworks like the cited USAE.

Taking this account and my limited expertise in this field, I will keep my Weak Accept.

Reviewer_sSy4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes translation from concept consistency, each at instance and distribution levels, and uses this framework to compare sparse autoencoder alignment methods. It introduces transfer-based evaluation metrics and a coupled SAE objective. Experiments cover vision, language, multimodal embeddings, and synthetic factors, with additional human evaluation and limited-pair training.

Critical strengths
  1. The separate evaluation of translation and concept consistency is useful. Table 2 shows a meaningful improvement over USAE in concept consistency while retaining comparable translation and extraction scores.
  2. The study combines real embeddings with controlled shared/private factors and human judgments. The limited-pair ablations also provide practical evidence that distributional regularization can help once paired examples are available.
Critical weaknesses
  1. The linear CCA claims do not hold under the stated assumptions. In Appendix J.3.2, lines 1844–1848, latent whitening does not make reconstruction constant: the relevant term becomes tr(WᵀΣ²W), rather than K. For example, let K=1, Σxx=Σyy=diag(1,100), and Σxy=diag(0.9,80). The leading CCA direction has consistency-plus-reconstruction loss 2.20 at λ=0.01, whereas the second, whitened direction achieves 0.42. The same example contradicts the CCA solution claimed for ordinary reduced-rank regression in Lemma 5. These errors affect the linear conclusions in Section 3.2 and Lemmas 4–6, although they do not by themselves invalidate the SAE experiments.
  2. The distributional concept loss does not identify the distributional property being defined. Equation 24 compares only coordinate marginals, whereas P_cont^dist requires equality of joint code distributions. Different sparse distributions can have identical coordinate marginals and zero loss. Independence of the underlying factors does not establish independence of learned codes. Consequently, unsuccessful instance alignment under this loss cannot distinguish ambiguity after successful joint matching from failure to match the joint distributions. This distinction matters for the paper's interpretation of the distributional experiments.
Questions for authors

What held-out joint-distribution discrepancy is achieved by the distributional-only models, and does using random concept-space projections change their instance-alignment results?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer sSy4 - Part I/II

We thank the reviewer for an exceptionally careful and technically precise review. The counterexample was very helpful, and we appreciate that the reviewer took the time to check the appendix proofs in this level of detail.


W1: Linear CCA claims. The reviewer is correct, and the counterexample is right. We went back through the proofs and identified two errors.

Error 1 : Lemma 4 (consistency + reconstruction).

  • Under latent whitening Wx⊤ΣxxWx=IKW_x^\top\Sigma_{xx}W_x = I_K, the projection is Px=WxWx⊤ΣxxP_x = W_xW_x^\top\Sigma_{xx}. This gives Tr(PxΣxx)=Tr(Wx⊤Σxx2Wx)\mathrm{Tr}(P_x\Sigma_{xx}) = \mathrm{Tr}(W_x^\top\Sigma_{xx}^2W_x), which is not the constant KK claimed in lines 1844–1848.
  • The reduced problem is therefore max⁡Wx,Wy2Tr(Wx⊤ΣxyWy)+λ(Tr(Wx⊤Σxx2Wx)+Tr(Wy⊤Σyy2Wy))\max_{W_x,W_y} 2\mathrm{Tr}(W_x^\top\Sigma_{xy}W_y) + \lambda\big(\mathrm{Tr}(W_x^\top\Sigma_{xx}^2W_x) + \mathrm{Tr}(W_y^\top\Sigma_{yy}^2W_y)\big) under the whitening constraints.
  • Its solution genuinely depends on λ\lambda. It interpolates between the CCA solution (λ→0\lambda \to 0) and the PCA solution (λ→∞\lambda \to \infty).
  • Additionally whitening the data (Σxx=Σyy=I\Sigma_{xx} = \Sigma_{yy} = I) would make the reconstruction term constant and restore the original statement. However, it trivializes almost everything: PCA becomes degenerate and CCA reduces to an SVD of Σxy\Sigma_{xy}. We therefore present the general interpolating solution, which is the more interesting case.

Error 2 : Lemma 5 (translation / RRR).

  • In the last step of the proof, the substitution C=DΣyy1/2C = D\Sigma_{yy}^{1/2} turns the Frobenius norm into a Mahalanobis-type norm weighted by Σyy\Sigma_{yy}, namely ∥(D−M)Σyy1/2∥F2\|(D - M)\Sigma_{yy}^{1/2}\|_F^2.
  • Eckart–Young does not apply to this weighted norm, so the truncated SVD of MM is not the minimizer in general. This is exactly the source of the discrepancy in the reviewer's example.
  • The correct solution is the standard RRR solution. It coincides with CCA under data whitening.
  • Lemma 6 is corrected accordingly: its solution remains the same as that of the corrected Lemma 5.

As the reviewer notes, none of the SAE experiments rely on this analysis. It was originally included for completeness. The consequences of this revision are as follows :

  • The corrected analysis is more informative. In the linear case, translation and concept consistency coupled with reconstruction lead to the same encoders only in the trivial whitened-data regime.
  • In general, once the properties are only approximately satisfied (here, traded off against reconstruction), the "dualities" already fail. This mirrors our SAE findings (Section 4.2) and strengthens the paper's main message.
  • We will rewrite item (B) of Section 3.2 and Lemmas 4–6 with their corrected proofs.

W2 and Q1: Distributional concept loss vs. PcontdistP^{\mathrm{dist}}_{\mathrm{cont}}. This is a valid point, and we thank the reviewer for spotting it.

  • The mismatch lies in the formulation of the property. PcontdistP^{\mathrm{dist}}_{\mathrm{cont}} is stated on the joint code distribution, whereas Equation 24 only matches coordinate marginals.
  • Hypothesis 2 assumes independent ground-truth concept coordinates (μ=∏kpk\mu = \prod_k p_k). Under that assumption, joint equality reduces to equality of marginals. We will rewrite PcontdistP^{\mathrm{dist}}_{\mathrm{cont}} accorcingly. However, as the reviewer rightly points out:
    • this independence is almost certainly false in practice;
    • independence of the underlying factors does not imply independence of the learned codes.
  • There is also a practical reason, which we should have stated plainly. The concept space is large (typically K=8dK = 8d) and sparse, so random projections in concept space carry very little information per slice and are expensive to compute. This is the honest reason why Lcontdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{cont}} is restricted to marginals empirically.

In the revision, we will:

  1. state the marginal version of the property PcontdistP^{\mathrm{dist}}_{\mathrm{cont}} that Lcontdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{cont}} actually operationalizes, together with the additional independence assumption under them;
  2. revise our interpretation of the distributional experiments. As the reviewer says, failure of instance alignment under this loss cannot distinguish ambiguity after successful joint matching from failure to match the joint distributions.
AuthorsOpenReview ↗

Rebuttal to Reviewer sSy4 - Part II/II

Q1: Held-out joint-distribution discrepancy of distributional-only models.

Ltrdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{tr}}.

  • Distributional matching is surprisingly effective, and even partially recovers instance-wise alignment.
  • In Table 1, line (c), Mtr=0.524M_{\mathrm{tr}} = 0.524 despite R2=0R^2 = 0. This means that at least for the atoms used by the sparse probes, alignment is meaningful.
  • Joint distributions nonetheless remain poorly aligned overall.
  • We are computing the held-out discrepancy (sliced Wasserstein, with null models as controls) and will post the exact values in a follow-up comment if the runs finish in time.

Lcontdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{cont}}.

  • Used alone, this loss has essentially no effect.
  • We believe the sparsity of concept activations makes it very hard to enforce without very large batch sizes.
  • joint vs marginal regularisation : in the former case, training is extremely unstable and leads to severely degenerate solutions, with no instance-level alignment recovered. Again because of sparsity, the loss is very noisy and uninformative.
  • We are happy to report this negative result in the appendix.

We thank the reviewer again for helping us correct and sharpen the theoretical part of the paper. We hope these answers and revisions address the concerns, and that the reviewer will consider raising their score.

Thank you for correcting the linear analysis. Section 4.2 still attributes the failure to recover instance-level alignment to non-identifiability. Since Eq. (24) compares coordinate marginals, the results leave open whether joint matching was achieved. I would ask that the manuscript reflect this distinction. I maintain my weak rejection recommendation.

Thank you for catching this and for your careful follow-up. Two changes were indeed missing from the main text in our previous revision:

  • Section 4.2 (lines 399–400). We now distinguish the two cases explicitly. For Ltrdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{tr}}, we attribute the failure to recover instance-level alignment to the non-identifiability of the transport problem. For Lcontdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{cont}}, we do not make this attribution: the loss compares only coordinate marginals, and the results therefore leave open whether joint matching was achieved.
  • Equation (6) (lines 280–285). The previous formulation still suggested joint distribution matching, with the distinction clarified only in the appendix. We now write Lcontdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{cont}} explicitly as an average of MMDs between coordinate marginals, with a comment below pointing to Appendix G for details.

We have also revised Appendix G (lines 1423–1426) to state explicitly that, under Hypothesis 2, PcontdistP^{\mathrm{dist}}_{\mathrm{cont}} reduces to marginal matching. We further clarify that independence is assumed for the learned codes rather than enforced by the loss. This distinction was not clear enough in the previous revision.


We hope this addresses your remaining concern. We are grateful for your detailed feedback throughout the discussion, which has helped us make these distinctions more precise.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall answer

We sincerely thank all three reviewers for their thorough, precise and constructive reviews. They identified real issues, including a technical error in our linear analysis (that does not impact the main results of the article) and several clarity problems, and the paper is clearly better for them. We are glad that the reviewers found the decomposition of alignment into translation and concept consistency useful (sSy4, mu66, EuUC), the empirical results strong (mu66), and the combination of real embeddings, controlled synthetic factors and human judgments convincing (sSy4).

Summary of changes in the revised version

  • Linear analysis (Section 3.2, Appendices E and J.3). Lemmas 4 and 5 are corrected, following Reviewer sSy4's counterexample. The corrected analysis shows that the translation/concept-consistency "duality" already breaks in the linear case once the properties are only approximately satisfied. This is consistent with, and strengthens, our main empirical message. The SAE experiments are unaffected.
  • Distributional concept consistency. We make explicit that Lcontdist\mathcal{L}^{\mathrm{dist}}_{\mathrm{cont}} matches coordinate marginals. We state the independence assumption under which this coincides with PcontdistP^{\mathrm{dist}}_{\mathrm{cont}}, and we revise the interpretation of the distributional experiments accordingly.
  • Notation. Following reviewers mu66 and EuUC’s advice, Equation (1) is renamed LCoSAE\mathcal{L}_{\mathrm{CoSAE}}. The standard SAE loss is explicitly identified as Ldcy\mathcal{L}_{\mathrm{dcy}}.
  • Hyperparameters. We add a practical recipe with the calibration rule (currently only in appendix) to the main body, the two observed regimes, and the reduced configuration (two hyperparameters) for fully paired settings.
  • Presentation.
    • Properties are named in the Figure 1 caption.
    • Table 1 now states metric directions and pairing proportions, and includes the full-objective row.
    • Figure 6 is revised to emphasize the comparison with Maniparambil et al. [64].
    • The limitations section is expanded.
    • The sliced ECF loss gets a code snippet in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.