- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper studies the concept alignment problem, i.e., measuring the consistency of the underlying concepts across two arbitrary representations (e.g., embeddings from different modalities or different models). The authors argue that ideal concept alignment should possess four indispensable properties, i.e., translation, concept consistency, instance-wise alignment, and distributional alignment. Based on such observations, the authors propose a compositional training objective for training Sparse Autoencoders (SAEs). To evaluate the quality of the trained SAEs, the authors further introduce the InterVenchA benchmark, which extends the SAEBench to assess translation quality and concept consistency.
Critical strengths
-
The theoretical analysis is solid. Based on the authors' discussion, using these four properties to evaluate concept agreement is reasonable, and each property is indispensable.
-
Introducing InterVenchA is reasonable. Translation and concept consistency are more informative than a single representation-similarity score.
Critical weaknesses
-
Due to the unclear presentation, the paper is difficult to follow. For example, 1) the symbols P_{dcy}, P_{tr}, P_{cont}, and P_{cy} that appear in Figure 1 are not explained in its caption or in the Introduction section. This confusion was not resolved until the Framework Formulation section. 2) The metrics in Table 1 should be annotated with whether higher values are better. 3) The regularization settings in Table 1 are also confusing. Variants (h)–(j) use paired data and specify the proportion, yet it is unclear how much paired data variants (b), (d), and (f) use, given that the L_{tr} and L_{count} losses are also based on paired data. 4) It is unclear why the results obtained with the full L_{SAE} are not reported in Table 1.
-
The main experiments aim to illustrate the effect of different regularization schemes on the quality of the trained SAEs. However, the weights of the different loss terms vary dramatically, e,g., α_{tr}=1 while α_{tr}^{dist}=1000. Such a large discrepancy makes variants (b) and (c) not directly comparable. A fair comparison of their contributions should be based on the optimal weight for each loss term.
-
In practice, applying Equation (1) to train the SAE raises the question of which regularization terms to retain and how to configure the weights, making the search space prohibitively large.
-
The authors need to demonstrate that the metrics (e.g., M_{tr} and M_{cont}) can reliably and correctly rank the relative concept consistency of different model pairs, rather than merely showing that a particular training objective yields higher scores. For example, given an anchor model A and two comparison models B and C, one needs to determine which of them has higher concept alignment with A. What matters here is discriminability rather than absolute values. Assume SAE1<A, B>=0.8, SAE1<A, C>=0.7, SAE2<A, B>=0.6, and SAE2<A, C>=0.1. Although SAE2 yields lower scores overall, it provides a clearer interpretation.
-
Does CoSAE refer to L_{SAE}? If so, why is Equation (1) not called L_{CoSAE}?
Questions for authors
Response to the questions in the weaknesses.
Rebuttal to Reviewer EuUC
We thank the reviewer for a thorough and careful review. The detailed list of clarity issues is extremely useful, and makes it clear that the presentation needs work. We address every point below.
W1: Clarity.
W2: Different weights across loss terms.
W3: Which terms to retain and how to configure them. We thank the reviewer for this comment. Identifying which terms matter and how to configure them is precisely one of the goals of the paper, but the reviewer's comment shows that we did not state this clearly enough. We will add the following sentence right after Equation (1):
W4: Discriminability across model pairs. This is a very interesting point, and we thank the reviewer for the discussion.
W5: CoSAE vs. LSAE. Yes, Equation (1) is the CoSAE objective. Other reviewers raised the same point, and we will rename it LCoSAE. Thank you for flagging this.
We are grateful for the reviewer's time and careful reading. We believe these changes substantially improve the paper's readability. We hope our responses address the concerns, and that the reviewer will consider raising their score.